[GitHub] [doris] BiteTheDDDDt merged pull request #12523: [Function](cbrt)Add cbrt function for doris
BiteThet merged PR #12523: URL: https://github.com/apache/doris/pull/12523 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt closed issue #12429: [Feature] The cbrt function support like hive.
BiteThet closed issue #12429: [Feature] The cbrt function support like hive. URL: https://github.com/apache/doris/issues/12429 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Function](cbrt)Add cbrt function for doris (#12523)
This is an automated email from the ASF dual-hosted git repository.
panxiaolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new c8e9a32bb2 [Function](cbrt)Add cbrt function for doris (#12523)
c8e9a32bb2 is described below
commit c8e9a32bb28297a80c37684982592fa525bcfb1a
Author: TaoZex <45089228+tao...@users.noreply.github.com>
AuthorDate: Mon Sep 12 19:58:45 2022 +0800
[Function](cbrt)Add cbrt function for doris (#12523)
Add cbrt function for doris
---
be/src/exprs/math_functions.cpp| 1 +
be/src/exprs/math_functions.h | 1 +
be/src/vec/functions/math.cpp | 8 ++-
be/test/vec/function/function_math_test.cpp| 11 +
.../sql-functions/math-functions/cbrt.md | 57 ++
.../sql-functions/math-functions/cbrt.md | 57 ++
gensrc/script/doris_builtins_functions.py | 2 +
7 files changed, 136 insertions(+), 1 deletion(-)
diff --git a/be/src/exprs/math_functions.cpp b/be/src/exprs/math_functions.cpp
index f21ebb2c62..3247f316c6 100644
--- a/be/src/exprs/math_functions.cpp
+++ b/be/src/exprs/math_functions.cpp
@@ -187,6 +187,7 @@ ONE_ARG_MATH_FN(acos, DoubleVal, DoubleVal, std::acos);
ONE_ARG_MATH_FN(tan, DoubleVal, DoubleVal, std::tan);
ONE_ARG_MATH_FN(atan, DoubleVal, DoubleVal, std::atan);
ONE_ARG_MATH_FN(sqrt, DoubleVal, DoubleVal, std::sqrt);
+ONE_ARG_MATH_FN(cbrt, DoubleVal, DoubleVal, std::cbrt);
ONE_ARG_MATH_FN(ceil, BigIntVal, DoubleVal, std::ceil);
ONE_ARG_MATH_FN(floor, BigIntVal, DoubleVal, std::floor);
ONE_ARG_MATH_FN(exp, DoubleVal, DoubleVal, std::exp);
diff --git a/be/src/exprs/math_functions.h b/be/src/exprs/math_functions.h
index edc4a394a7..9ac791ab61 100644
--- a/be/src/exprs/math_functions.h
+++ b/be/src/exprs/math_functions.h
@@ -88,6 +88,7 @@ public:
const doris_udf::DoubleVal& v);
static doris_udf::DoubleVal sqrt(doris_udf::FunctionContext*, const
doris_udf::DoubleVal&);
+static doris_udf::DoubleVal cbrt(doris_udf::FunctionContext*, const
doris_udf::DoubleVal&);
static doris_udf::DoubleVal pow(doris_udf::FunctionContext* ctx,
const doris_udf::DoubleVal& base,
const doris_udf::DoubleVal& exp);
diff --git a/be/src/vec/functions/math.cpp b/be/src/vec/functions/math.cpp
index 6278d33567..417f576a52 100644
--- a/be/src/vec/functions/math.cpp
+++ b/be/src/vec/functions/math.cpp
@@ -266,6 +266,11 @@ struct SqrtName {
};
using FunctionSqrt = FunctionMathUnary>;
+struct CbrtName {
+static constexpr auto name = "cbrt";
+};
+using FunctionCbrt = FunctionMathUnary>;
+
struct TanName {
static constexpr auto name = "tan";
};
@@ -444,6 +449,7 @@ void register_function_math(SimpleFunctionFactory& factory)
{
factory.register_function();
factory.register_function();
factory.register_alias("sqrt", "dsqrt");
+factory.register_function();
factory.register_function();
factory.register_function();
factory.register_alias("floor", "dfloor");
@@ -460,4 +466,4 @@ void register_function_math(SimpleFunctionFactory& factory)
{
factory.register_function();
factory.register_function();
}
-} // namespace doris::vectorized
\ No newline at end of file
+} // namespace doris::vectorized
diff --git a/be/test/vec/function/function_math_test.cpp
b/be/test/vec/function/function_math_test.cpp
index 940bbc995e..a93fb02603 100644
--- a/be/test/vec/function/function_math_test.cpp
+++ b/be/test/vec/function/function_math_test.cpp
@@ -108,6 +108,17 @@ TEST(MathFunctionTest, sqrt_test) {
check_function(func_name, input_types, data_set);
}
+TEST(MathFunctionTest, cbrt_test) {
+std::string func_name = "cbrt";
+
+InputTypeSet input_types = {TypeIndex::Float64};
+
+DataSet data_set = {
+{{0.0}, 0.0}, {{2.0}, 1.2599210498948734}, {{8.0}, 2.0},
{{-1000.0}, -10.0}};
+
+check_function(func_name, input_types, data_set);
+}
+
TEST(MathFunctionTest, tan_test) {
std::string func_name = "tan"; //tan(x)
diff --git a/docs/en/docs/sql-manual/sql-functions/math-functions/cbrt.md
b/docs/en/docs/sql-manual/sql-functions/math-functions/cbrt.md
new file mode 100644
index 00..24fcba137e
--- /dev/null
+++ b/docs/en/docs/sql-manual/sql-functions/math-functions/cbrt.md
@@ -0,0 +1,57 @@
+---
+{
+"title": "cbrt",
+"language": "en"
+}
+---
+
+
+
+## cbrt
+
+### description
+ Syntax
+
+`DOUBLE cbrt(DOUBLE x)`
+Returns the cube root of x.
+
+### example
+
+```
+mysql> select cbrt(8);
++---+
+| cbrt(8.0) |
++---+
+| 2 |
++---+
+mysql> select cbrt(2.0);
+++
+| cbrt(2.0) |
+++
+| 1.2599210498948734 |
+++
+mysql> select cbrt(-1
[GitHub] [doris-website] ZHbamboo opened a new pull request, #93: [doc]: Add blog 'Doris analysis: Doris Compaction mechanism analysis'
ZHbamboo opened a new pull request, #93: URL: https://github.com/apache/doris-website/pull/93 # Changes Add blog 'Doris analysis: Doris Compaction mechanism analysis' in both Chinese and English language with pictures. Issue Number: #11706 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #10322: [feature](JSONB datatype)Support JSONB datatype
xiaokang commented on code in PR #10322:
URL: https://github.com/apache/doris/pull/10322#discussion_r967566758
##
fe/fe-core/src/main/java/org/apache/doris/mysql/MysqlColType.java:
##
@@ -51,7 +51,8 @@ public enum MysqlColType {
MYSQL_TYPE_BLOB(252, "BLOB"),
MYSQL_TYPE_VARSTRING(253, "VAR STRING"),
MYSQL_TYPE_STRING(254, "STRING"),
-MYSQL_TYPE_GEOMETRY(255, "GEOMETRY");
+MYSQL_TYPE_GEOMETRY(255, "GEOMETRY"),
+MYSQL_TYPE_JSONB(256, "JSONB");
Review Comment:
refer to the mysql link above
##
fe/fe-core/src/main/java/org/apache/doris/catalog/Type.java:
##
@@ -148,6 +149,19 @@ public abstract class Type {
supportedTypes.add(HLL);
supportedTypes.add(BITMAP);
supportedTypes.add(QUANTILE_STATE);
+supportedTypes.add(CHAR);
Review Comment:
why we add these types here?
##
fe/fe-core/src/main/java/org/apache/doris/catalog/PrimitiveType.java:
##
@@ -49,6 +49,7 @@ public enum PrimitiveType {
// 8-byte pointer and 4-byte length indicator (12 bytes total).
// Aligning to 8 bytes so 16 total.
VARCHAR("VARCHAR", 16, TPrimitiveType.VARCHAR),
+JSONB("JSONB", 16, TPrimitiveType.JSONB),
Review Comment:
we need to consider it carefully
##
be/test/runtime/jsonb_value_test.cpp:
##
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "runtime/jsonb_value.h"
+
+#include
+
+#include
+
+#include "util/cpu_info.h"
+
+using std::string;
+
+namespace doris {
+
+JsonBinaryValue FromStdString(const string& str) {
+char* ptr = const_cast(str.c_str());
+int len = str.size();
+return JsonBinaryValue(ptr, len);
+}
+
+TEST(JsonBinaryValueTest, TestValidation) {
+JsonbErrType err;
+JsonBinaryValue json_val;
+
+// single value not wrapped as an arrar or object is invalid
+std::vector invalid_strs = {"", "1", "null", "false", "abc"};
+for (size_t i = 0; i < invalid_strs.size(); i++) {
+err = json_val.from_json_str(invalid_strs[i].c_str(),
invalid_strs[i].size());
+EXPECT_NE(err, JsonbErrType::E_NONE);
+std::cout << JsonbErrMsg::getErrMsg(err) << std::endl;
Review Comment:
not necessary
##
gensrc/thrift/Exprs.thrift:
##
@@ -40,6 +40,7 @@ enum TExprNodeType {
NULL_LITERAL,
SLOT_REF,
STRING_LITERAL,
+ JSONB_LITERAL,
Review Comment:
json_literal is better, since literal is json format
##
be/src/vec/exprs/vliteral.cpp:
##
@@ -231,6 +240,11 @@ std::string VLiteral::debug_string() const {
out << ref;
break;
}
+case TYPE_JSONB: {
+JsonBinaryValue value(ref.data, ref.size);
Review Comment:
not necessary, just out << ref
##
be/src/runtime/jsonb_value.h:
##
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#ifndef DORIS_BE_RUNTIME_JSON_VALUE_H
+#define DORIS_BE_RUNTIME_JSON_VALUE_H
+
+#include "udf/udf.h"
+#include "util/cpu_info.h"
+#include "util/hash_util.hpp"
+#include "util/jsonb_error.h"
+#include "util/jsonb_parser.h"
+#include "util/jsonb_utils.h"
+#include "vec/common/string_ref.h"
+
+#ifdef __SSE4_2__
+#include "util/sse_util.hpp"
+#endif
+
+namespace doris {
+
+struct JsonBinaryValue {
Review Comment:
JsonbValue is enough
##
be/src/vec/sink/vmysql_result_writer.cpp:
##
@@ -106,6 +106,21 @@ Status VM
[GitHub] [doris] zy-kkk commented on pull request #12529: [test](window-function) add regression test of window function
zy-kkk commented on PR #12529: URL: https://github.com/apache/doris/pull/12529#issuecomment-1243883755 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TaoZex opened a new pull request, #12532: [DOCS](function) Add docs of math function
TaoZex opened a new pull request, #12532: URL: https://github.com/apache/doris/pull/12532 # Proposed changes Issue Number: ## Problem summary I found that a lot of math function documentation was missing, so I filled it in. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [x] No - [ ] No Need 3. Has document been added or modified: - [x] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wsjz opened a new pull request, #12533: [feature-wip](parquet-reader) add page index
wsjz opened a new pull request, #12533: URL: https://github.com/apache/doris/pull/12533 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #12507: [Bug] be may core dump when load column mapping has function
yiguolei closed issue #12507: [Bug] be may core dump when load column mapping has function URL: https://github.com/apache/doris/issues/12507 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12509: [Bugfix](load) fix be may core dump when load column mapping has function
yiguolei merged PR #12509: URL: https://github.com/apache/doris/pull/12509 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Bugfix](load) fix be may core dump when load column mapping has function (#12509)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 503a79e4d8 [Bugfix](load) fix be may core dump when load column
mapping has function (#12509)
503a79e4d8 is described below
commit 503a79e4d824c2846b5b8957c4331ba2d4d2f3e5
Author: Zhengguo Yang
AuthorDate: Tue Sep 13 08:44:10 2022 +0800
[Bugfix](load) fix be may core dump when load column mapping has function
(#12509)
fix be may core dump when load column mapping has function
this bug may be introduced by #12375
---
.../doris/load/loadv2/LoadingTaskPlanner.java | 36 --
1 file changed, 20 insertions(+), 16 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadingTaskPlanner.java
b/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadingTaskPlanner.java
index 3796040f21..dedd436a78 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadingTaskPlanner.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadingTaskPlanner.java
@@ -109,29 +109,33 @@ public class LoadingTaskPlanner {
throws UserException {
// Generate tuple descriptor
TupleDescriptor destTupleDesc = descTable.createTupleDescriptor();
-TupleDescriptor scanTupleDesc =
descTable.createTupleDescriptor("ScanTuple");
+TupleDescriptor scanTupleDesc = destTupleDesc;
+if (Config.enable_vectorized_load) {
+scanTupleDesc = descTable.createTupleDescriptor("ScanTuple");
+}
// use full schema to fill the descriptor table
for (Column col : table.getFullSchema()) {
SlotDescriptor slotDesc =
descTable.addSlotDescriptor(destTupleDesc);
slotDesc.setIsMaterialized(true);
slotDesc.setColumn(col);
slotDesc.setIsNullable(col.isAllowNull());
-
-SlotDescriptor scanSlotDesc =
descTable.addSlotDescriptor(scanTupleDesc);
-scanSlotDesc.setIsMaterialized(true);
-scanSlotDesc.setColumn(col);
-scanSlotDesc.setIsNullable(col.isAllowNull());
-if (fileGroups.size() > 0) {
-for (ImportColumnDesc importColumnDesc :
fileGroups.get(0).getColumnExprList()) {
-try {
-if (!importColumnDesc.isColumn() &&
importColumnDesc.getColumnName() != null
-&&
importColumnDesc.getColumnName().equals(col.getName())) {
-
scanSlotDesc.setIsNullable(importColumnDesc.getExpr().isNullable());
-break;
+if (Config.enable_vectorized_load) {
+SlotDescriptor scanSlotDesc =
descTable.addSlotDescriptor(scanTupleDesc);
+scanSlotDesc.setIsMaterialized(true);
+scanSlotDesc.setColumn(col);
+scanSlotDesc.setIsNullable(col.isAllowNull());
+if (fileGroups.size() > 0) {
+for (ImportColumnDesc importColumnDesc :
fileGroups.get(0).getColumnExprList()) {
+try {
+if (!importColumnDesc.isColumn() &&
importColumnDesc.getColumnName() != null
+&&
importColumnDesc.getColumnName().equals(col.getName())) {
+
scanSlotDesc.setIsNullable(importColumnDesc.getExpr().isNullable());
+break;
+}
+} catch (Exception e) {
+// An exception may be thrown here because the
`importColumnDesc.getExpr()` is not analyzed
+// now. We just skip this case here.
}
-} catch (Exception e) {
-// An exception may be thrown here because the
`importColumnDesc.getExpr()` is not analyzed
-// now. We just skip this case here.
}
}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on pull request #12512: [Enhancement](execute) add timeout for executing fragment rpc
yiguolei commented on PR #12512: URL: https://github.com/apache/doris/pull/12512#issuecomment-1244753846 Could you explain the reason why need add timeout for the rpc in Problem Summary? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Bug](array_type) Forbid adding array key columns #12479
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b1c2a8343f [Bug](array_type) Forbid adding array key columns #12479
b1c2a8343f is described below
commit b1c2a8343f429baebf5aa5cc6e7dec7749983bab
Author: xy720 <22125576+xy...@users.noreply.github.com>
AuthorDate: Tue Sep 13 08:48:28 2022 +0800
[Bug](array_type) Forbid adding array key columns #12479
mysql> desc array_test;
+---++--+---+-+---+
| Field | Type | Null | Key | Default | Extra |
+---++--+---+-+---+
| id| INT| Yes | true | NULL| |
| c_array | ARRAY | Yes | false | NULL| NONE |
+---++--+---+-+---+
Before:
mysql> ALTER TABLE array_test ADD COLUMN add_arr_key array key NULL
DEFAULT NULL;
Query OK, 0 rows affected (0.00 sec)
After:
mysql> ALTER TABLE array_test ADD COLUMN c_array array key NULL
DEFAULT NULL;
ERROR 1105 (HY000): errCode = 2, detailMessage = Array can only be used in
the non-key column of the duplicate table at present.
mysql> ALTER TABLE array_test MODIFY COLUMN c_array array key NULL
DEFAULT NULL;
ERROR 1105 (HY000): errCode = 2, detailMessage = Array can only be used in
the non-key column of the duplicate table at present.
---
fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java | 4
1 file changed, 4 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
index 788b5e59f1..5a944aa7b2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ColumnDef.java
@@ -265,6 +265,10 @@ public class ColumnDef {
}
if (type.getPrimitiveType() == PrimitiveType.ARRAY) {
+if (isKey()) {
+throw new AnalysisException("Array can only be used in the
non-key column of"
++ " the duplicate table at present.");
+}
if (defaultValue.isSet && defaultValue !=
DefaultValue.NULL_DEFAULT_VALUE) {
throw new AnalysisException("Array type column default value
only support null");
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12479: [Bug](array_type) Forbid adding array key columns
yiguolei merged PR #12479: URL: https://github.com/apache/doris/pull/12479 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei closed issue #12412: [Bug] too many wait_for_start() thread and can not execute query
yiguolei closed issue #12412: [Bug] too many wait_for_start() thread and can not execute query URL: https://github.com/apache/doris/issues/12412 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12411: [fix](exec) Avoid query thread block on wait_for_start
yiguolei merged PR #12411: URL: https://github.com/apache/doris/pull/12411 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](exec) Avoid query thread block on wait_for_start (#12411)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new e33f4f90ae [fix](exec) Avoid query thread block on wait_for_start
(#12411)
e33f4f90ae is described below
commit e33f4f90aed728c343757fcba3cc99aa4459185e
Author: Mingyu Chen
AuthorDate: Tue Sep 13 08:57:37 2022 +0800
[fix](exec) Avoid query thread block on wait_for_start (#12411)
When FE send cancel rpc to BE, it does not notify the wait_for_start()
thread, so that the fragment will be blocked and occupy the execution thread.
Add a max wait time for wait_for_start() thread. So that it will not block
forever.
---
be/src/common/config.h | 8 +++-
be/src/runtime/fragment_mgr.cpp| 48 ++
be/src/runtime/plan_fragment_executor.cpp | 8 ++--
be/src/runtime/plan_fragment_executor.h| 5 ---
be/src/runtime/query_fragments_ctx.h | 13 --
be/src/util/doris_metrics.h| 1 +
be/src/vec/exec/volap_scan_node.cpp| 5 +--
be/test/runtime/fragment_mgr_test.cpp | 8
docs/sidebars.json | 1 -
.../maint-monitor/monitor-metrics/metrics.md | 1 +
10 files changed, 46 insertions(+), 52 deletions(-)
diff --git a/be/src/common/config.h b/be/src/common/config.h
index a099de3c4f..cf24e4de8b 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -859,6 +859,13 @@ CONF_Bool(enable_new_scan_node, "true");
// limit the queue of pending batches which will be sent by a single
nodechannel
CONF_mInt64(nodechannel_pending_queue_max_bytes, "67108864");
+// Max waiting time to wait the "plan fragment start" rpc.
+// If timeout, the fragment will be cancelled.
+// This parameter is usually only used when the FE loses connection,
+// and the BE can automatically cancel the relevant fragment after the timeout,
+// so as to avoid occupying the execution thread for a long time.
+CONF_mInt32(max_fragment_start_wait_time_seconds, "30");
+
#ifdef BE_TEST
// test s3
CONF_String(test_s3_resource, "resource");
@@ -869,7 +876,6 @@ CONF_String(test_s3_region, "region");
CONF_String(test_s3_bucket, "bucket");
CONF_String(test_s3_prefix, "prefix");
#endif
-
} // namespace config
} // namespace doris
diff --git a/be/src/runtime/fragment_mgr.cpp b/be/src/runtime/fragment_mgr.cpp
index c5b3cbf818..1b86f57a94 100644
--- a/be/src/runtime/fragment_mgr.cpp
+++ b/be/src/runtime/fragment_mgr.cpp
@@ -60,6 +60,7 @@ namespace doris {
DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(plan_fragment_count, MetricUnit::NOUNIT);
DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(timeout_canceled_fragment_count,
MetricUnit::NOUNIT);
+DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(fragment_thread_pool_queue_size,
MetricUnit::NOUNIT);
std::string to_load_error_http_path(const std::string& file_name) {
if (file_name.empty()) {
@@ -95,8 +96,6 @@ public:
Status execute();
-Status cancel_before_execute();
-
Status cancel(const PPlanFragmentCancelReason& reason, const std::string&
msg = "");
TUniqueId fragment_instance_id() const { return _fragment_instance_id; }
@@ -236,13 +235,20 @@ Status FragmentExecState::execute() {
if (_need_wait_execution_trigger) {
// if _need_wait_execution_trigger is true, which means this instance
// is prepared but need to wait for the signal to do the rest
execution.
-_fragments_ctx->wait_for_start();
-opentelemetry::trace::Tracer::GetCurrentSpan()->AddEvent("start
executing Fragment");
+if (!_fragments_ctx->wait_for_start()) {
+return cancel(PPlanFragmentCancelReason::INTERNAL_ERROR, "wait
fragment start timeout");
+}
+}
+#ifndef BE_TEST
+if (_executor.runtime_state()->is_cancelled()) {
+return Status::Cancelled("cancelled before execution");
}
+#endif
int64_t duration_ns = 0;
{
SCOPED_RAW_TIMER(&duration_ns);
CgroupsMgr::apply_system_cgroup();
+opentelemetry::trace::Tracer::GetCurrentSpan()->AddEvent("start
executing Fragment");
WARN_IF_ERROR(_executor.open(), strings::Substitute("Got error while
opening fragment $0",
print_id(_fragment_instance_id)));
@@ -253,24 +259,9 @@ Status FragmentExecState::execute() {
return Status::OK();
}
-Status FragmentExecState::cancel_before_execute() {
-// set status as 'abort', cuz cancel() won't effect the status arg of
DataSink::close().
-#ifndef BE_TEST
-SCOPED_ATTACH_TASK(executor()->runtime_state());
-#endif
-_executor.set_abort();
-_executor.cancel();
-if (_pipe != nullptr) {
-_pipe->cancel("Execution aborted before start");
-}
-return Status::OK();
-}
-
Status Fragmen
[GitHub] [doris] yiguolei merged pull request #12529: [test](window-function) add regression test of window function
yiguolei merged PR #12529: URL: https://github.com/apache/doris/pull/12529 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [test](window-function) add regression test of window function (#12529)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 4c73755b40 [test](window-function) add regression test of window function (#12529) 4c73755b40 is described below commit 4c73755b40ee4ac7f14b21920fb91b93842cfb4c Author: lsy3993 <110876560+lsy3...@users.noreply.github.com> AuthorDate: Tue Sep 13 08:58:19 2022 +0800 [test](window-function) add regression test of window function (#12529) --- .../window_functions/test_window_fn.out| 221 + .../window_functions/test_window_fn.groovy | 128 2 files changed, 349 insertions(+) diff --git a/regression-test/data/query_p0/sql_functions/window_functions/test_window_fn.out b/regression-test/data/query_p0/sql_functions/window_functions/test_window_fn.out new file mode 100644 index 00..efef9420e4 --- /dev/null +++ b/regression-test/data/query_p0/sql_functions/window_functions/test_window_fn.out @@ -0,0 +1,221 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !sql -- +3500 0 6000 +3500 39003500 +3500 42003900 +3500 45004200 +3500 48004500 +3500 48004800 +3500 50004800 +3500 52005000 +3500 52005200 +3500 60005200 + +-- !sql -- +6000 450035002007-12-10 +6000 450039002006-12-23 +6000 450042002008-01-01 +6000 450045002008-01-01 +6000 450048002007-08-01 +6000 450048002007-08-08 +6000 450050002006-10-01 +6000 450052002007-08-01 +6000 450052002007-08-15 +6000 450060002006-10-01 + +-- !sql -- +6000 0 3500 +6000 35003900 +6000 39004200 +6000 42004500 +6000 45004800 +6000 48004800 +6000 48005000 +6000 50005200 +6000 52005200 +6000 52006000 + +-- !sql -- +1 sales 50001000200 1000200 +2 personnel 39001000200 1000200 +3 sales 4800500 \N 500 200 +4 sales 4800500 \N 500 200 +5 personnel 3500500 \N 500 200 +7 develop 4200\N \N 500 200 +8 develop 60001000200 500 200 +9 develop 4500\N \N 500 200 +10 develop 5200500 200 500 200 +11 develop 5200500 200 500 200 + +-- !sql -- +2008-01-01 35002007-12-10 +2008-01-01 39002006-12-23 +2008-01-01 42002008-01-01 +2008-01-01 45002008-01-01 +2008-01-01 48002007-08-01 +2008-01-01 48002007-08-08 +2008-01-01 50002006-10-01 +2008-01-01 52002007-08-01 +2008-01-01 52002007-08-15 +2008-01-01 60002006-10-01 + +-- !sql -- +2008-01-01 35002007-12-10 +2008-01-01 39002006-12-23 +2008-01-01 42002008-01-01 +2008-01-01 45002008-01-01 +2008-01-01 48002007-08-01 +2008-01-01 48002007-08-08 +2008-01-01 50002006-10-01 +2008-01-01 52002007-08-01 +2008-01-01 52002007-08-15 +2008-01-01 60002006-10-01 + +-- !sql -- +2008-01-01 35002007-12-10 +2008-01-01 39002006-12-23 +2008-01-01 42002008-01-01 +2008-01-01 45002008-01-01 +2008-01-01 48002007-08-01 +2008-01-01 48002007-08-08 +2008-01-01 50002006-10-01 +2008-01-01 52002007-08-01 +2008-01-01 52002007-08-15 +2008-01-01 60002006-10-01 + +-- !sql -- +develop7 42001 +develop9 45002 +develop10 52003 +develop11 52003 +develop8 60005 +personnel 5 35001 +personnel 2 39002 +sales 3 48001 +sales 4 48001 +sales 1 50003 + +-- !sql -- +develop7 42001 +personnel 5 35001 +sales 3 48001 +develop9 45002 +personnel 2 39002 +sales 4 48002 +develop10 52003 +sales 1 50003 +develop11 52004 +develop8 60005 + +-- !sql -- +7400 2 22000 +14600 3 14600 +25100 1 47100 + +-- !sql -- +3900 1 +5000 1 +6000 1 +7400 2 +14600 2 +14600 2 +16400 2 +16400 2 +20900 4 +25100 5 + +-- !sql -- +personnel 5 2007-12-10 35001 +personnel 2 2006-12-23 39002 +develop7 2008-01-01 42003 +develop9 2008-01-01 45004 +sales 3 2007-08-01 48005 +sales 4 2007-08-08 48006 +sales 1 2006-10-01 50007 +develop10 2007-08-01 52008 +develop11 2007-08-15 52009 + +-- !sql -- +develo
[doris] branch master updated: [test](join)add some join cases (#12501)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 8be5527be4 [test](join)add some join cases (#12501) 8be5527be4 is described below commit 8be5527be436c5b27526eaed159551ef6e72097d Author: zy-kkk AuthorDate: Tue Sep 13 08:59:32 2022 +0800 [test](join)add some join cases (#12501) --- regression-test/data/query/join/test_join2.out | 236 + .../suites/query/join/test_join2.groovy| 161 ++ 2 files changed, 397 insertions(+) diff --git a/regression-test/data/query/join/test_join2.out b/regression-test/data/query/join/test_join2.out new file mode 100644 index 00..6ec7da8cdb --- /dev/null +++ b/regression-test/data/query/join/test_join2.out @@ -0,0 +1,236 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !join1 -- + 0 \N zero0 \N + 1 4 one 1 -1 + 2 3 two 2 2 + 2 3 two 2 4 + 3 2 three 3 -3 + 5 0 five5 -5 + 5 0 five5 -5 + +-- !join2 -- + 0 \N zero0 \N + 1 4 one 1 -1 + 2 3 two 2 2 + 2 3 two 2 4 + 3 2 three 3 -3 + 5 0 five5 -5 + 5 0 five5 -5 + +-- !join3 -- + \N \N null\N \N + \N \N null\N 0 + \N \N null0 \N + \N \N null1 -1 + \N \N null2 2 + \N \N null2 4 + \N \N null3 -3 + \N \N null5 -5 + \N \N null5 -5 + \N 0 zero\N \N + \N 0 zero\N 0 + \N 0 zero0 \N + \N 0 zero1 -1 + \N 0 zero2 2 + \N 0 zero2 4 + \N 0 zero3 -3 + \N 0 zero5 -5 + \N 0 zero5 -5 + 0 \N zero\N \N + 0 \N zero\N 0 + 0 \N zero0 \N + 0 \N zero1 -1 + 0 \N zero2 2 + 0 \N zero2 4 + 0 \N zero3 -3 + 0 \N zero5 -5 + 0 \N zero5 -5 + 1 4 one \N \N + 1 4 one \N 0 + 1 4 one 0 \N + 1 4 one 1 -1 + 1 4 one 2 2 + 1 4 one 2 4 + 1 4 one 3 -3 + 1 4 one 5 -5 + 1 4 one 5 -5 + 2 3 two \N \N + 2 3 two \N 0 + 2 3 two 0 \N + 2 3 two 1 -1 + 2 3 two 2 2 + 2 3 two 2 4 + 2 3 two 3 -3 + 2 3 two 5 -5 + 2 3 two 5 -5 + 3 2 three \N \N + 3 2 three \N 0 + 3 2 three 0 \N + 3 2 three 1 -1 + 3 2 three 2 2 + 3 2 three 2 4 + 3 2 three 3 -3 + 3 2 three 5 -5 + 3 2 three 5 -5 + 4 1 four\N \N + 4 1 four\N 0 + 4 1 four0 \N + 4 1 four1 -1 + 4 1 four2 2 + 4 1 four2 4 + 4 1 four3 -3 + 4 1 four5 -5 + 4 1 four5 -5 + 5 0 five\N \N + 5 0 five\N 0 + 5 0 five0 \N + 5 0 five1 -1 + 5 0 five2 2 + 5 0 five2 4 + 5 0 five3 -3 + 5 0 five5 -5 + 5 0 five5 -5 + 6 6 six \N \N + 6 6 six \N 0 + 6 6 six 0 \N + 6 6 six 1 -1 + 6 6 six 2 2 + 6 6 six 2 4 +
[GitHub] [doris] yiguolei merged pull request #12501: [test](join)add join case2
yiguolei merged PR #12501: URL: https://github.com/apache/doris/pull/12501 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12508: [test](join)add test join case4
yiguolei merged PR #12508: URL: https://github.com/apache/doris/pull/12508 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [test](join)add test join case4 #12508
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 97cb095010 [test](join)add test join case4 #12508
97cb095010 is described below
commit 97cb0950101d792b397a0b156e449165017a0953
Author: zy-kkk
AuthorDate: Tue Sep 13 09:09:49 2022 +0800
[test](join)add test join case4 #12508
---
regression-test/data/query/join/test_join4.out | 58 +
.../suites/query/join/test_join4.groovy| 59 ++
2 files changed, 117 insertions(+)
diff --git a/regression-test/data/query/join/test_join4.out
b/regression-test/data/query/join/test_join4.out
new file mode 100644
index 00..a332202a4b
--- /dev/null
+++ b/regression-test/data/query/join/test_join4.out
@@ -0,0 +1,58 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !join1 --
+1 11 1 111
+2 22 2 222
+3 \N \N \N
+4 44 4 \N
+5 \N \N \N
+
+-- !join2 --
+1 11 1 111
+2 22 2 222
+3 \N 3 333
+4 44 \N \N
+5 \N \N \N
+
+-- !join3 --
+1 11 1 111 1 11
+2 22 2 222 2 22
+3 \N 3 333 3 \N
+4 44 4 \N 4 44
+5 \N \N \N 5 \N
+
+-- !join4 --
+1 11 1 111 1 11
+2 22 2 222 2 22
+3 \N 3 333 \N \N
+4 44 4 \N 4 44
+5 \N \N \N \N \N
+
+-- !join5 --
+1 11 1 111 1 11
+2 22 2 222 2 22
+3 \N 3 333 3 \N
+4 44 4 \N \N \N
+5 \N \N \N \N \N
+
+-- !join6 --
+1 11 1 111 1 11
+2 22 2 222 2 22
+3 \N 3 333 \N \N
+4 44 4 \N 4 44
+5 \N \N \N \N \N
+
+-- !join7 --
+1 11 1 111 1 11
+2 22 2 222 2 22
+4 44 4 \N 4 44
+
+-- !join8 --
+1 11 1 111 1 11
+2 22 2 222 2 22
+3 \N 3 333 3 \N
+
+-- !join9 --
+1 11 1 111 1 11
+2 22 2 222 2 22
+4 44 4 \N 4 44
+
diff --git a/regression-test/suites/query/join/test_join4.groovy
b/regression-test/suites/query/join/test_join4.groovy
new file mode 100644
index 00..41838d554f
--- /dev/null
+++ b/regression-test/suites/query/join/test_join4.groovy
@@ -0,0 +1,59 @@
+ // Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_join4", "query,p0") {
+def DBname = "test_join4"
+sql "DROP DATABASE IF EXISTS ${DBname}"
+sql "CREATE DATABASE IF NOT EXISTS ${DBname}"
+sql "use ${DBname}"
+
+def tbName1 = "x"
+def tbName2 = "y"
+
+sql "DROP TABLE IF EXISTS ${tbName1};"
+sql "DROP TABLE IF EXISTS ${tbName2};"
+
+sql """create table ${tbName1} (x1 int, x2 int) DISTRIBUTED BY HASH(x1)
properties("replication_num" = "1");"""
+sql """create table ${tbName2} (y1 int, y2 int) DISTRIBUTED BY HASH(y1)
properties("replication_num" = "1");"""
+
+sql "insert into ${tbName1} values (1,11);"
+sql "insert into ${tbName1} values (2,22);"
+sql "insert into ${tbName1} values (3,null);"
+sql "insert into ${tbName1} values (4,44);"
+sql "insert into ${tbName1} values (5,null);"
+
+sql "insert into ${tbName2} values (1,111);"
+sql "insert into ${tbName2} values (2,222);"
+sql "insert into ${tbName2} values (3,333);"
+sql "insert into ${tbName2} values (4,null);"
+
+qt_join1 "select * from ${tbName1} left join ${tbName2} on (x1 = y1 and x2
is not null) order by 1,2,3,4;"
+qt_join2 "select * from ${tbName1} left join ${tbName2} on (x1 = y1 and y2
is not null) ord
[GitHub] [doris] stalary commented on a diff in pull request #12401: [Enhancement](DOE): Doe support object/nested use string
stalary commented on code in PR #12401: URL: https://github.com/apache/doris/pull/12401#discussion_r969052909 ## be/src/http/http_client.cpp: ## @@ -162,7 +163,8 @@ Status HttpClient::execute(const std::function
[GitHub] [doris] stalary commented on issue #12528: [Bug]
stalary commented on issue #12528: URL: https://github.com/apache/doris/issues/12528#issuecomment-1244779174 Clean fe meta and try to restart it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary commented on issue #12525: [Feature] Support the use of group_concat function and the use of separator
stalary commented on issue #12525: URL: https://github.com/apache/doris/issues/12525#issuecomment-1244782209 What version is used? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary commented on issue #12454: [Bug] Communication failure occurred during jdbc connection usage
stalary commented on issue #12454: URL: https://github.com/apache/doris/issues/12454#issuecomment-1244784889 Can you submit your fe log? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary closed issue #3250: BE 启动后，执行查询或其它操作自动挂掉
stalary closed issue #3250: BE 启动后，执行查询或其它操作自动挂掉 URL: https://github.com/apache/doris/issues/3250 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary closed issue #3023: doris can support to limit the timeout of one sql execution
stalary closed issue #3023: doris can support to limit the timeout of one sql execution URL: https://github.com/apache/doris/issues/3023 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary closed issue #3084: mac how to connect docker BE
stalary closed issue #3084: mac how to connect docker BE URL: https://github.com/apache/doris/issues/3084 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-website] hf200012 merged pull request #92: [typo] Correct Chinese text error
hf200012 merged PR #92: URL: https://github.com/apache/doris-website/pull/92 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: typo (#92)
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 0aaa5d4c1be typo (#92) 0aaa5d4c1be is described below commit 0aaa5d4c1be8a137bb58affad090eecaf31941b3 Author: TaoZex <45089228+tao...@users.noreply.github.com> AuthorDate: Tue Sep 13 09:53:05 2022 +0800 typo (#92) --- .../docusaurus-plugin-content-docs/current/get-starting/get-starting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/get-starting/get-starting.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/get-starting/get-starting.md index e07ddf3dd2d..8de78be4911 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/get-starting/get-starting.md +++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/get-starting/get-starting.md @@ -324,7 +324,7 @@ PROPERTIES ( curl --location-trusted -u root: -T test.csv -H "column_separator:," http://127.0.0.1:8030/api/demo/expamle_tbl/_stream_load ``` -- -T test.csv : 这里使我们刚才保存的数据文件，如果路径不一样，请指定完整路径 +- -T test.csv : 这里是我们刚才保存的数据文件，如果路径不一样，请指定完整路径 - -u root : 这里是用户名密码，我们使用默认用户root，密码是空 - 127.0.0.1:8030 : 分别是 fe 的 ip 和 http_port - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12401: [Enhancement](DOE): Doe support object/nested use string
github-actions[bot] commented on PR #12401: URL: https://github.com/apache/doris/pull/12401#issuecomment-1244799796 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary merged pull request #12401: [Enhancement](DOE): Doe support object/nested use string
stalary merged PR #12401: URL: https://github.com/apache/doris/pull/12401 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary closed issue #12387: [Enhancement] Doe support query object/nested data
stalary closed issue #12387: [Enhancement] Doe support query object/nested data URL: https://github.com/apache/doris/issues/12387 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Enhancement](DOE): Doe support object/nested use string (#12401)
This is an automated email from the ASF dual-hosted git repository.
stalary pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 87439e227e [Enhancement](DOE): Doe support object/nested use string
(#12401)
87439e227e is described below
commit 87439e227e02ad1903b9c926e5eb5d2ff3e42eed
Author: Stalary
AuthorDate: Tue Sep 13 09:59:48 2022 +0800
[Enhancement](DOE): Doe support object/nested use string (#12401)
* MOD: doe support object/nested use string
---
be/src/http/http_client.cpp| 4 ++-
.../doris/external/elasticsearch/EsUtil.java | 37 +++---
.../doris/external/elasticsearch/MappingPhase.java | 3 ++
.../doris/external/elasticsearch/EsUtilTest.java | 8 +
.../test/resources/data/es/test_index_mapping.json | 30 ++
.../data/es/test_index_mapping_after_7x.json | 30 ++
.../es/test_index_mapping_field_mult_analyzer.json | 30 ++
7 files changed, 123 insertions(+), 19 deletions(-)
diff --git a/be/src/http/http_client.cpp b/be/src/http/http_client.cpp
index 96d9c447e4..0cb5b97ec4 100644
--- a/be/src/http/http_client.cpp
+++ b/be/src/http/http_client.cpp
@@ -18,6 +18,7 @@
#include "http/http_client.h"
#include "common/config.h"
+#include "util/stack_util.h"
namespace doris {
@@ -162,7 +163,8 @@ Status HttpClient::execute(const std::function
[GitHub] [doris] github-actions[bot] commented on pull request #12505: [regression](json) add a nullable case for stream load with json format
github-actions[bot] commented on PR #12505: URL: https://github.com/apache/doris/pull/12505#issuecomment-1244806226 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12505: [regression](json) add a nullable case for stream load with json format
github-actions[bot] commented on PR #12505: URL: https://github.com/apache/doris/pull/12505#issuecomment-1244806257 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12316: [Enhancement](compaction) reduce VMergeIterator copy block
github-actions[bot] commented on PR #12316: URL: https://github.com/apache/doris/pull/12316#issuecomment-1244808961 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12316: [Enhancement](compaction) reduce VMergeIterator copy block
github-actions[bot] commented on PR #12316: URL: https://github.com/apache/doris/pull/12316#issuecomment-1244808945 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12527: [feature-wip](new-scan) refactor some interface about predicate push down in scan node
github-actions[bot] commented on PR #12527: URL: https://github.com/apache/doris/pull/12527#issuecomment-1244809588 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12527: [feature-wip](new-scan) refactor some interface about predicate push down in scan node
github-actions[bot] commented on PR #12527: URL: https://github.com/apache/doris/pull/12527#issuecomment-1244809557 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BenjaminWenqiYu closed pull request #12225: [enhancement]MySQL ODBC should use a variable driver without a fixed value
BenjaminWenqiYu closed pull request #12225: [enhancement]MySQL ODBC should use a variable driver without a fixed value URL: https://github.com/apache/doris/pull/12225 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #12527: [feature-wip](new-scan) refactor some interface about predicate push down in scan node
morningman merged PR #12527: URL: https://github.com/apache/doris/pull/12527 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature-wip](new-scan) refactor some interface about predicate push down in scan node (#12527)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8a274d7851 [feature-wip](new-scan) refactor some interface about
predicate push down in scan node (#12527)
8a274d7851 is described below
commit 8a274d7851b2007c03d31a60b8fedbb329dc8f82
Author: Mingyu Chen
AuthorDate: Tue Sep 13 10:25:13 2022 +0800
[feature-wip](new-scan) refactor some interface about predicate push down
in scan node (#12527)
This PR introduce a new enum type `PushDownType`:
```
enum class PushDownType {
// The predicate can not be pushed down to data source
UNACCEPTABLE,
// The predicate can be pushed down to data source
// and the data source can fully evaludate it
ACCEPTABLE,
// The predicate can be pushed down to data source
// but the data source can not fully evaluate it.
PARTIAL_ACCEPTABLE
};
```
And derived class of VScanNode can override following method to determine
whether to accept
a bianry/in/bloom filter/is null predicate:
```
PushDownType _should_push_down_binary_predicate();
PushDownType _should_push_down_in_predicate();
PushDownType _should_push_down_function_filter();
PushDownType _should_push_down_bloom_filter();
PushDownType _should_push_down_is_null_predicate();
```
---
be/src/vec/exec/scan/new_olap_scan_node.cpp | 77 ++---
be/src/vec/exec/scan/new_olap_scan_node.h | 14 +-
be/src/vec/exec/scan/vscan_node.cpp | 259 ++--
be/src/vec/exec/scan/vscan_node.h | 59 +--
4 files changed, 224 insertions(+), 185 deletions(-)
diff --git a/be/src/vec/exec/scan/new_olap_scan_node.cpp
b/be/src/vec/exec/scan/new_olap_scan_node.cpp
index d4b1b6a7d7..973e6c23ee 100644
--- a/be/src/vec/exec/scan/new_olap_scan_node.cpp
+++ b/be/src/vec/exec/scan/new_olap_scan_node.cpp
@@ -125,7 +125,7 @@ static std::string olap_filters_to_string(const
std::vector&
filters_string += "[";
for (auto it = filters.cbegin(); it != filters.cend(); it++) {
if (it != filters.cbegin()) {
-filters_string += ",";
+filters_string += ", ";
}
filters_string += olap_filter_to_string(*it);
}
@@ -181,6 +181,12 @@ Status NewOlapScanNode::_build_key_ranges_and_filters() {
}
}
+// Append value ranges in "_not_in_value_ranges"
+for (auto& range : _not_in_value_ranges) {
+std::visit([&](auto&& the_range) {
the_range.to_in_condition(_olap_filters, false); },
+ range);
+}
+
_runtime_profile->add_info_string("PushDownPredicates",
olap_filters_to_string(_olap_filters));
_runtime_profile->add_info_string("KeyRanges", _scan_keys.debug_string());
VLOG_CRITICAL << _scan_keys.debug_string();
@@ -188,67 +194,12 @@ Status NewOlapScanNode::_build_key_ranges_and_filters() {
return Status::OK();
}
-bool NewOlapScanNode::_should_push_down_binary_predicate(
-VectorizedFnCall* fn_call, VExprContext* expr_ctx, StringRef*
constant_val,
-int* slot_ref_child, const std::function&
fn_checker) {
-if (!fn_checker(fn_call->fn().name.function_name)) {
-return false;
-}
-
-const auto& children = fn_call->children();
-DCHECK(children.size() == 2);
-for (size_t i = 0; i < children.size(); i++) {
-if (VExpr::expr_without_cast(children[i])->node_type() !=
TExprNodeType::SLOT_REF) {
-// not a slot ref(column)
-continue;
-}
-if (!children[1 - i]->is_constant()) {
-// only handle constant value
-return false;
-} else {
-if (const ColumnConst* const_column =
check_and_get_column(
-children[1 - i]->get_const_col(expr_ctx)->column_ptr))
{
-*slot_ref_child = i;
-*constant_val = const_column->get_data_at(0);
-} else {
-return false;
-}
-}
-}
-return true;
-}
-
-bool NewOlapScanNode::_should_push_down_in_predicate(VInPredicate* pred,
VExprContext* expr_ctx,
- bool is_not_in) {
-if (pred->is_not_in() != is_not_in) {
-return false;
-}
-InState* state = reinterpret_cast(
-expr_ctx->fn_context(pred->fn_context_index())
-->get_function_state(FunctionContext::FRAGMENT_LOCAL));
-HybridSetBase* set = state->hybrid_set.get();
-
-// if there are too many elements in InPredicate, exceed the limit,
-// we will not push any condition of this column to storage engine.
-// because too many conditions pushed down to storage engine may even
-// slow down the query process.
-// ATT
[doris] branch master updated (8a274d7851 -> 9f25544f2f)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 8a274d7851 [feature-wip](new-scan) refactor some interface about predicate push down in scan node (#12527) add 9f25544f2f [feature-wip](parquet-reader) page index bug fix (#12428) No new revisions were added by this update. Summary of changes: .../exec/format/parquet/vparquet_page_index.cpp| 2 ++ be/src/vec/exec/format/parquet/vparquet_reader.cpp | 35 +++--- 2 files changed, 19 insertions(+), 18 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #12428: [feature-wip](parquet-reader) page index bug fix
morningman merged PR #12428: URL: https://github.com/apache/doris/pull/12428 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhangstar333 opened a new pull request, #12534: [Feature](vectorized) support jdbc sink for insert into data to table
zhangstar333 opened a new pull request, #12534: URL: https://github.com/apache/doris/pull/12534 # Proposed changes support jdbc sink for insert into data to table `insert into jdbc_table select * from table` ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [regression](json) add a nullable case for stream load with json format (#12505)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 353f9e3782 [regression](json) add a nullable case for stream load with
json format (#12505)
353f9e3782 is described below
commit 353f9e37825d135dee11b9993b8529fbda2c7ed7
Author: yinzhijian <373141...@qq.com>
AuthorDate: Tue Sep 13 10:45:01 2022 +0800
[regression](json) add a nullable case for stream load with json format
(#12505)
---
.../stream_load/load_json_null_to_nullable.out | 13 +++
.../data/load_p0/stream_load/test_char.json| 9 ++
.../stream_load/load_json_null_to_nullable.groovy | 109 +
3 files changed, 131 insertions(+)
diff --git
a/regression-test/data/load_p0/stream_load/load_json_null_to_nullable.out
b/regression-test/data/load_p0/stream_load/load_json_null_to_nullable.out
new file mode 100644
index 00..d564b5b99a
--- /dev/null
+++ b/regression-test/data/load_p0/stream_load/load_json_null_to_nullable.out
@@ -0,0 +1,13 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !select --
+\N \N
+
+H H
+h h
+
+-- !select --
+\N \N
+
+H H
+h h
+
diff --git a/regression-test/data/load_p0/stream_load/test_char.json
b/regression-test/data/load_p0/stream_load/test_char.json
new file mode 100644
index 00..7cd7d69d9b
--- /dev/null
+++ b/regression-test/data/load_p0/stream_load/test_char.json
@@ -0,0 +1,9 @@
+[
+{"k1": "h", "v1": "h"},
+{"k1": "hello", "v1": "hello"},
+{"k1": "hello,hello", "v1": "hello,hello"},
+{"k1": "仓库", "v1": "安全"},
+{"k1": "H", "v1": "H"},
+{"k1": "", "v1": ""},
+{"k1": null, "v1": null}
+]
diff --git
a/regression-test/suites/load_p0/stream_load/load_json_null_to_nullable.groovy
b/regression-test/suites/load_p0/stream_load/load_json_null_to_nullable.groovy
new file mode 100644
index 00..e9a5665e03
--- /dev/null
+++
b/regression-test/suites/load_p0/stream_load/load_json_null_to_nullable.groovy
@@ -0,0 +1,109 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_load_json_null_to_nullable", "p0") {
+// define a sql table
+def testTable = "tbl_test_json_load"
+def dbName = "test_query_db"
+
+def create_test_table = {enable_vectorized_flag ->
+if (enable_vectorized_flag) {
+sql """ set enable_vectorized_engine = true """
+} else {
+sql """ set enable_vectorized_engine = false """
+}
+
+def result1 = sql """
+CREATE TABLE IF NOT EXISTS ${testTable} (
+ `k1` CHAR NULL COMMENT "",
+ `v1` CHAR NULL COMMENT ""
+) ENGINE=OLAP
+DUPLICATE KEY(`k1`)
+DISTRIBUTED BY HASH(`k1`) BUCKETS 1
+PROPERTIES (
+"replication_allocation" = "tag.location.default: 1",
+"storage_format" = "V2"
+)
+"""
+}
+
+def load_array_data = {table_name, strip_flag, read_flag, format_flag,
exprs, json_paths,
+json_root, where_expr, fuzzy_flag, column_sep,
file_name ->
+// load the json data
+streamLoad {
+table table_name
+
+// set http request header params
+set 'strip_outer_array', strip_flag
+set 'read_json_by_line', read_flag
+set 'format', format_flag
+set 'columns', exprs
+set 'jsonpaths', json_paths
+set 'json_root', json_root
+set 'where', where_expr
+set 'fuzzy_parse', fuzzy_flag
+set 'column_separator', column_sep
+set 'max_filter_ratio', '1'
+file file_name // import json file
+time 1 // limit inflight 10s
+
+// if declared a check callback, the default check condition will
ignore.
+// So you must check all condition
+check { result, exception, startTime, endTime ->
+if (exception != null) {
+t
[GitHub] [doris] morningman merged pull request #12505: [regression](json) add a nullable case for stream load with json format
morningman merged PR #12505: URL: https://github.com/apache/doris/pull/12505 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12424: [fix](array-type) fix the invalid format load for stream load
github-actions[bot] commented on PR #12424: URL: https://github.com/apache/doris/pull/12424#issuecomment-1244829624 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12424: [fix](array-type) fix the invalid format load for stream load
github-actions[bot] commented on PR #12424: URL: https://github.com/apache/doris/pull/12424#issuecomment-1244829654 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 opened a new pull request, #12535: [enhancement](agg)remove unnessasery mem alloc and dealloc in agg node
starocean999 opened a new pull request, #12535: URL: https://github.com/apache/doris/pull/12535 # Proposed changes Issue Number: close #xxx ## Problem summary remove unnessasery mem alloc and dealloc in agg node ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener merged pull request #12500: [fix](doc) add the key columes description of the table model document
jackwener merged PR #12500: URL: https://github.com/apache/doris/pull/12500 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (353f9e3782 -> 550b1e531b)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 353f9e3782 [regression](json) add a nullable case for stream load with json format (#12505) add 550b1e531b [fix](doc) add the key columes description of the table model document (#12500) No new revisions were added by this update. Summary of changes: docs/en/docs/data-table/data-model.md| 3 +++ docs/zh-CN/docs/data-table/data-model.md | 3 +++ 2 files changed, 6 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #12441: [fix](agg)the intermediate slots should be materialized as output slots
yiguolei merged PR #12441: URL: https://github.com/apache/doris/pull/12441 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (550b1e531b -> 6b52e47805)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 550b1e531b [fix](doc) add the key columes description of the table
model document (#12500)
add 6b52e47805 [fix](agg)the intermediate slots should be materialized as
output slots (#12441)
No new revisions were added by this update.
Summary of changes:
.../org/apache/doris/analysis/AggregateInfo.java | 24
.../apache/doris/analysis/AggregateInfoBase.java | 11
.../org/apache/doris/analysis/DescriptorTable.java | 13
.../main/java/org/apache/doris/analysis/Expr.java | 12
.../org/apache/doris/analysis/SlotDescriptor.java | 1 +
.../org/apache/doris/planner/AggregationNode.java | 4 +-
.../apache/doris/planner/SingleNodePlanner.java| 1 +
.../test_subquery_with_agg.out}| 0
.../correctness_p0/test_subquery_with_agg.groovy | 70 ++
9 files changed, 135 insertions(+), 1 deletion(-)
copy regression-test/data/{query_p0/intersect/test_intersect.out =>
correctness_p0/test_subquery_with_agg.out} (100%)
create mode 100644
regression-test/suites/correctness_p0/test_subquery_with_agg.groovy
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12470: [fix](frontend) fix notify update storage policy agent task null exception
github-actions[bot] commented on PR #12470: URL: https://github.com/apache/doris/pull/12470#issuecomment-1244853416 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12470: [fix](frontend) fix notify update storage policy agent task null exception
github-actions[bot] commented on PR #12470: URL: https://github.com/apache/doris/pull/12470#issuecomment-1244853435 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 commented on a diff in pull request #10084: [feature](vectorized) Support block aggregate in scanner
Gabriel39 commented on code in PR #10084:
URL: https://github.com/apache/doris/pull/10084#discussion_r969110789
##
be/src/util/simd/bits.h:
##
@@ -89,16 +92,31 @@ inline size_t count_zero_num(const int8_t* __restrict data,
size_t size) {
return num;
}
-inline size_t count_zero_num(const int8_t* __restrict data, const uint8_t*
__restrict null_map,
Review Comment:
why remove `__restrict` ?
##
be/src/vec/columns/column_decimal.cpp:
##
@@ -240,13 +240,13 @@ ColumnPtr ColumnDecimal::filter(const IColumn::Filter&
filt, ssize_t result_s
* completely pass or do not pass the filter.
* Therefore, we will optimistically check the parts of `SIMD_BYTES`
values.
*/
-static constexpr size_t SIMD_BYTES = 32;
+static constexpr size_t SIMD_BYTES = 64;
const UInt8* filt_end_sse = filt_pos + size / SIMD_BYTES * SIMD_BYTES;
while (filt_pos < filt_end_sse) {
-uint32_t mask = simd::bytes32_mask_to_bits32_mask(filt_pos);
+auto mask = simd::bytes64_mask_to_bits64_mask(filt_pos);
-if (0x == mask) {
+if (0x == mask) {
Review Comment:
Could you make this mask to a common constexpr? I see it is used in multiple
places.
##
be/src/util/simd/bits.h:
##
@@ -89,16 +92,31 @@ inline size_t count_zero_num(const int8_t* __restrict data,
size_t size) {
return num;
}
-inline size_t count_zero_num(const int8_t* __restrict data, const uint8_t*
__restrict null_map,
Review Comment:
This function seems never used. Could we remove it?
##
be/src/vec/olap/block_reader.cpp:
##
@@ -168,6 +177,49 @@ Status BlockReader::init(const ReaderParams& read_params) {
return Status::OK();
}
+Status BlockReader::next_block_with_aggregation(Block* block, MemPool*
mem_pool,
+ObjectPool* agg_pool, bool*
eof) {
+return _block_aggregator ? _agg_next_block(block, mem_pool, agg_pool, eof)
Review Comment:
If agg is push down, we'd better to let `_next_block_func` be
`BlockReader::_agg_next_block` instead of make this condition judgment each time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #12534: [Feature](vectorized) support jdbc sink for insert into data to table
HappenLee commented on code in PR #12534:
URL: https://github.com/apache/doris/pull/12534#discussion_r969128036
##
be/src/vec/exec/vjdbc_connector.cpp:
##
@@ -301,6 +321,118 @@ Status JdbcConnector::_convert_column_data(JNIEnv* env,
jobject jobj,
return Status::OK();
}
+Status JdbcConnector::append(const std::string& table_name, vectorized::Block*
block,
+ const std::vector&
_output_vexpr_ctxs,
+ uint32_t start_send_row, uint32_t* num_rows_sent)
{
+_insert_stmt_buffer.clear();
+std::u16string insert_stmt;
+{
+SCOPED_TIMER(_convert_tuple_timer);
+fmt::format_to(_insert_stmt_buffer, "INSERT INTO {} VALUES (",
table_name);
+
+int num_rows = block->rows();
+int num_columns = block->columns();
+for (int i = start_send_row; i < num_rows; ++i) {
+(*num_rows_sent)++;
+
+// Construct insert statement of jdbc table
+for (int j = 0; j < num_columns; ++j) {
+if (j != 0) {
+fmt::format_to(_insert_stmt_buffer, "{}", ", ");
+}
+auto [item, size] =
block->get_by_position(j).column->get_data_at(i);
+if (item == nullptr) {
+fmt::format_to(_insert_stmt_buffer, "{}", "NULL");
+continue;
+}
+switch (_output_vexpr_ctxs[j]->root()->type().type) {
+case TYPE_BOOLEAN:
+case TYPE_TINYINT:
+fmt::format_to(_insert_stmt_buffer, "{}",
+ *reinterpret_cast(item));
+break;
+case TYPE_SMALLINT:
+fmt::format_to(_insert_stmt_buffer, "{}",
+ *reinterpret_cast(item));
+break;
+case TYPE_INT:
+fmt::format_to(_insert_stmt_buffer, "{}",
+ *reinterpret_cast(item));
+break;
+case TYPE_BIGINT:
+fmt::format_to(_insert_stmt_buffer, "{}",
+ *reinterpret_cast(item));
+break;
+case TYPE_FLOAT:
+fmt::format_to(_insert_stmt_buffer, "{}",
+ *reinterpret_cast(item));
+break;
+case TYPE_DOUBLE:
+fmt::format_to(_insert_stmt_buffer, "{}",
+ *reinterpret_cast(item));
+break;
+case TYPE_DATE:
Review Comment:
support `NEW_DATE/NEW_DECIMAL`
##
be/src/vec/exec/vjdbc_connector.cpp:
##
@@ -301,6 +321,118 @@ Status JdbcConnector::_convert_column_data(JNIEnv* env,
jobject jobj,
return Status::OK();
}
+Status JdbcConnector::append(const std::string& table_name, vectorized::Block*
block,
+ const std::vector&
_output_vexpr_ctxs,
+ uint32_t start_send_row, uint32_t* num_rows_sent)
{
+_insert_stmt_buffer.clear();
+std::u16string insert_stmt;
Review Comment:
this code should be ABTRUCT in a single class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener merged pull request #12506: [fix](Nereids): fix LAsscom project split.
jackwener merged PR #12506: URL: https://github.com/apache/doris/pull/12506 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (6b52e47805 -> c3d7d4ce7a)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 6b52e47805 [fix](agg)the intermediate slots should be materialized as
output slots (#12441)
add c3d7d4ce7a [fix](Nereids): fix LAsscom project split. (#12506)
No new revisions were added by this update.
Summary of changes:
.../rules/exploration/join/JoinLAsscomHelper.java | 14 +++-
.../rules/exploration/join/ThreeJoinHelper.java| 38 ++---
.../apache/doris/nereids/util/ExpressionUtils.java | 16 +++-
...AsscomTest.java => JoinLAsscomProjectTest.java} | 93 --
.../rules/exploration/join/JoinLAsscomTest.java| 15 +---
.../nereids/util/AnalyzeWhereSubqueryTest.java | 12 +++
.../apache/doris/nereids/util/PlanUtilsTest.java | 60 ++
7 files changed, 143 insertions(+), 105 deletions(-)
copy
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/join/{JoinLAsscomTest.java
=> JoinLAsscomProjectTest.java} (52%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/util/PlanUtilsTest.java
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #12511: [feature](Nereids): Eliminate redundant filter and limit.
morrySnow commented on code in PR #12511:
URL: https://github.com/apache/doris/pull/12511#discussion_r969132655
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/BooleanLiteral.java:
##
@@ -32,11 +32,27 @@ public class BooleanLiteral extends Literal {
private final boolean value;
-public BooleanLiteral(boolean value) {
+private BooleanLiteral(boolean value) {
super(BooleanType.INSTANCE);
this.value = value;
}
+public static BooleanLiteral of(boolean value) {
+if (value) {
+return TRUE;
+} else {
+return FALSE;
+}
+}
+
+public static BooleanLiteral of(Boolean value) {
Review Comment:
i think we do not need this function, since java could do auto unboxing
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/EliminateFilter.java:
##
@@ -0,0 +1,39 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.plans.logical.LogicalEmptyRelation;
+
+import java.util.List;
+
+/**
+ * Eliminate filter false.
+ */
+public class EliminateFilter extends OneRewriteRuleFactory {
+@Override
+public Rule build() {
+return logicalFilter()
+.when(filter -> filter.getPredicates() == BooleanLiteral.FALSE)
+.then(limit -> new LogicalEmptyRelation((List)
limit.getOutput()))
Review Comment:
```suggestion
.then(filter -> new LogicalEmptyRelation((List)
filter.getOutput()))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei opened a new pull request, #12538: [do not merge 1.1.2] ADD PROFILE IN SEGMENT ITERATGOR
yiguolei opened a new pull request, #12538: URL: https://github.com/apache/doris/pull/12538 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #12539: [fix](Nereids): fix StatsCalculator compute project.
jackwener opened a new pull request, #12539: URL: https://github.com/apache/doris/pull/12539 # Proposed changes Issue Number: close #xxx ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 4. Does it need to update dependencies: - [ ] Yes - [x] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #12490: [feature](nereids) push down Project through Limit
morrySnow merged PR #12490: URL: https://github.com/apache/doris/pull/12490 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature](nereids) push down Project through Limit (#12490)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d35a8a24a5 [feature](nereids) push down Project through Limit (#12490)
d35a8a24a5 is described below
commit d35a8a24a5fc03e6dcdb17cde4bd3a76ec888710
Author: Kikyou1997 <33112463+kikyou1...@users.noreply.github.com>
AuthorDate: Tue Sep 13 13:26:12 2022 +0800
[feature](nereids) push down Project through Limit (#12490)
This rule is rewrite project -> limit to limit -> project. The reason is we
could get tree like project -> limit -> project -> other node. If we do not
rewrite it. we could not merge the two project into one. And if we has more
than one project on one node, the second one will overwrite the first one when
translate. Then, be will core dump or return slot cannot find error.
---
.../doris/nereids/jobs/batch/RewriteJob.java | 7 +++-
.../org/apache/doris/nereids/rules/RuleType.java | 1 +
...ject.java => PushdownFilterThroughProject.java} | 2 +-
...oject.java => PushdownProjectThroughLimit.java} | 46
.../logical/PushdownProjectThroughLimitTest.java | 49 ++
5 files changed, 84 insertions(+), 21 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/RewriteJob.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/RewriteJob.java
index 5ec029005b..e8d347000d 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/RewriteJob.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/batch/RewriteJob.java
@@ -31,8 +31,9 @@ import
org.apache.doris.nereids.rules.rewrite.logical.MergeConsecutiveProjects;
import org.apache.doris.nereids.rules.rewrite.logical.NormalizeAggregate;
import org.apache.doris.nereids.rules.rewrite.logical.PruneOlapScanPartition;
import org.apache.doris.nereids.rules.rewrite.logical.PushPredicateThroughJoin;
+import
org.apache.doris.nereids.rules.rewrite.logical.PushdownFilterThroughProject;
+import
org.apache.doris.nereids.rules.rewrite.logical.PushdownProjectThroughLimit;
import org.apache.doris.nereids.rules.rewrite.logical.ReorderJoin;
-import org.apache.doris.nereids.rules.rewrite.logical.SwapFilterAndProject;
import com.google.common.collect.ImmutableList;
@@ -54,6 +55,7 @@ public class RewriteJob extends BatchRulesJob {
* 1. Adjust the plan in correlated logicalApply
*so that there are no correlated columns in the subquery.
* 2. Convert logicalApply to a logicalJoin.
+ * TODO: group these rules to make sure the result plan is
what we expected.
*/
.addAll(new
AdjustApplyFromCorrelatToUnCorrelatJob(cascadesContext).rulesJob)
.addAll(new ConvertApplyToJoinJob(cascadesContext).rulesJob)
@@ -65,7 +67,8 @@ public class RewriteJob extends BatchRulesJob {
.add(topDownBatch(ImmutableList.of(new NormalizeAggregate(
.add(topDownBatch(ImmutableList.of(new ColumnPruning(
.add(topDownBatch(ImmutableList.of(new
AggregateDisassemble(
-.add(topDownBatch(ImmutableList.of(new
SwapFilterAndProject(
+.add(topDownBatch(ImmutableList.of(new
PushdownProjectThroughLimit(
+.add(topDownBatch(ImmutableList.of(new
PushdownFilterThroughProject(
.add(bottomUpBatch(ImmutableList.of(new
MergeConsecutiveProjects(
.add(topDownBatch(ImmutableList.of(new
MergeConsecutiveFilters(
.add(bottomUpBatch(ImmutableList.of(new
MergeConsecutiveLimits(
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
index 31a028eb9e..78b2a42d33 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/RuleType.java
@@ -105,6 +105,7 @@ public enum RuleType {
OLAP_SCAN_PARTITION_PRUNE(RuleTypeClass.REWRITE),
SWAP_FILTER_AND_PROJECT(RuleTypeClass.REWRITE),
LOGICAL_LIMIT_TO_LOGICAL_EMPTY_RELATION_RULE(RuleTypeClass.REWRITE),
+SWAP_LIMIT_PROJECT(RuleTypeClass.REWRITE),
// exploration rules
TEST_EXPLORATION(RuleTypeClass.EXPLORATION),
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/SwapFilterAndProject.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushdownFilterThroughProject.java
similarity index 96%
copy from
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/SwapFilterAndProject.java
copy to
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushdownFilterThroughProject.java
ind
[GitHub] [doris] jackwener commented on a diff in pull request #12511: [feature](Nereids): Eliminate redundant filter and limit.
jackwener commented on code in PR #12511:
URL: https://github.com/apache/doris/pull/12511#discussion_r969165351
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/BooleanLiteral.java:
##
@@ -32,11 +32,27 @@ public class BooleanLiteral extends Literal {
private final boolean value;
-public BooleanLiteral(boolean value) {
+private BooleanLiteral(boolean value) {
super(BooleanType.INSTANCE);
this.value = value;
}
+public static BooleanLiteral of(boolean value) {
+if (value) {
+return TRUE;
+} else {
+return FALSE;
+}
+}
+
+public static BooleanLiteral of(Boolean value) {
Review Comment:
Literal also has `of` function, if this don't exist, `Literal of` will call
itself which is dead loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener merged pull request #12515: [feature](Nereids): semi join transpose.
jackwener merged PR #12515: URL: https://github.com/apache/doris/pull/12515 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated (d35a8a24a5 -> 5b4d3616a4)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from d35a8a24a5 [feature](nereids) push down Project through Limit (#12490) add 5b4d3616a4 [feature](Nereids): semi join transpose. (#12515) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/nereids/memo/Memo.java | 10 ++ .../join/SemiJoinLogicalJoinTranspose.java | 100 + .../join/SemiJoinLogicalJoinTransposeProject.java | 122 + .../join/SemiJoinSemiJoinTranspose.java| 78 + .../exploration/join/SemiJoinTransposeTest.java| 63 +++ .../doris/nereids/util/LogicalPlanBuilder.java | 78 + 6 files changed, 451 insertions(+) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinLogicalJoinTranspose.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinLogicalJoinTransposeProject.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinSemiJoinTranspose.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/rules/exploration/join/SemiJoinTransposeTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/util/LogicalPlanBuilder.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #12275: [feature-wip](new-scan) New load scanner.
morningman merged PR #12275: URL: https://github.com/apache/doris/pull/12275 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
svn commit: r56797 - /dev/doris/1.1.2-rc05/ /release/doris/1.1/1.1.2-rc05/
Author: morningman Date: Tue Sep 13 05:42:19 2022 New Revision: 56797 Log: move doris 1.1.2-rc05 to release Added: release/doris/1.1/1.1.2-rc05/ - copied from r56796, dev/doris/1.1.2-rc05/ Removed: dev/doris/1.1.2-rc05/ - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hopepanwei opened a new issue, #12540: [Bug] CSV uses a custom separator, and the number of fields cannot be correctly recognized
hopepanwei opened a new issue, #12540: URL: https://github.com/apache/doris/issues/12540 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.1.1 release ### What's Wrong? Reason: actual column number is less than schema column number.actual number: 22, column separator: [||,], line delimiter: [ ], schema number: 23; . src line [***||,1||,2019-07-12 17:32:08.320216||,2020-08-03 12:58:14.419272||,1990-04-24 21:52:40.00||,**||,4||,||,**||,*||, ||,*||,货币||,400||,2010-04-08 00:00:00||,1|||,400||,2010-04-08 00:00:00||,1||,15240726||,15454216||,*||,]; I used | |, as the delimiter, and this data cutting reported an error This data should be 23 fields after segmentation, but only 22 are identified ### What You Expected? How to identify the correct number of fields ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BenjaminWenqiYu opened a new issue, #12541: [Bug] fix mysql_to_doris user_define_tables files path
BenjaminWenqiYu opened a new issue, #12541: URL: https://github.com/apache/doris/issues/12541 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version v1.1.0 ### What's Wrong? For the mysql to doris extension, the user defined tables shell should create sqls to the path user_files. ### What You Expected? For the mysql to doris extension, the user defined tables shell should create sqls to the path user_files. ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TaoZex commented on pull request #12532: [DOCS](function) Add docs of math function
TaoZex commented on PR #12532: URL: https://github.com/apache/doris/pull/12532#issuecomment-1244944959 Do I need to submit again in doris-website? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BenjaminWenqiYu opened a new pull request, #12542: Update user_define_tables.sh
BenjaminWenqiYu opened a new pull request, #12542: URL: https://github.com/apache/doris/pull/12542 # Proposed changes Issue Number: close #12541 ## Problem summary For the mysql to doris extension, the user defined tables shell should create sqls to the path user_files. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] nextdreamblue commented on pull request #12513: [feature](http) refactor version info and add new http api for get version info
nextdreamblue commented on PR #12513: URL: https://github.com/apache/doris/pull/12513#issuecomment-1244947183 > It is recommended that the current version can be viewed in the start script of fe and be For example. sh bin/start_fe.sh --version sh bin/start_be.sh --version if client want get version info, the script is not simple to use. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] 01/01: create branch 1.1 lts
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch branch-1.1-lts in repository https://gitbox.apache.org/repos/asf/doris.git commit 214ccb02c4ca8041c326c0ca2e075ea6eb254c50 Author: morningman AuthorDate: Tue Sep 13 14:18:38 2022 +0800 create branch 1.1 lts --- gensrc/script/gen_build_version.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gensrc/script/gen_build_version.sh b/gensrc/script/gen_build_version.sh index 8f8d96a3c3..4dd3f44472 100755 --- a/gensrc/script/gen_build_version.sh +++ b/gensrc/script/gen_build_version.sh @@ -25,7 +25,7 @@ # contains the build version based on the git hash or svn revision. ## -build_version="1.1.2-rc05" +build_version="1.1-lts" unset LANG unset LC_CTYPE - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-1.1-lts created (now 214ccb02c4)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch branch-1.1-lts in repository https://gitbox.apache.org/repos/asf/doris.git at 214ccb02c4 create branch 1.1 lts This branch includes the following new commits: new 214ccb02c4 create branch 1.1 lts The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] carlvinhust2012 opened a new pull request, #12543: [fix](array-type) forbid to create materialized view for array column
carlvinhust2012 opened a new pull request, #12543: URL: https://github.com/apache/doris/pull/12543 # Proposed changes 1. this pr is used to forbid to create materialized view for array column. 2. before the change Issue Number: #7570 ## Problem summary Describe your changes. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BenjaminWenqiYu opened a new issue, #12544: [Bug] Fix the date-time-functions doc.
BenjaminWenqiYu opened a new issue, #12544: URL: https://github.com/apache/doris/issues/12544 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version v1.1.* ### What's Wrong? should use CURRENT_DATE to obtain the correct result ### What You Expected? should use CURRENT_DATE to obtain the correct result ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BenjaminWenqiYu opened a new pull request, #12545: Fix sql function doc
BenjaminWenqiYu opened a new pull request, #12545: URL: https://github.com/apache/doris/pull/12545 # Proposed changes should use CURRENT_DATE to obtain the correct result Issue Number: close #12544 ## Problem summary should use CURRENT_DATE to obtain the correct result. ## Checklist(Required) 1. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 2. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 3. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 4. Does it need to update dependencies: - [ ] Yes - [ ] No 5. Are there any changes that cannot be rolled back: - [ ] Yes (If Yes, please explain WHY) - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
