[doris] branch master updated: [fix](be): fix stack overflow in unhex function (#11204)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b260a02215 [fix](be): fix stack overflow in unhex function (#11204)
b260a02215 is described below
commit b260a02215033e87c67b2ae09ba2ac9ff0a50aef
Author: spaces-x
AuthorDate: Thu Jul 28 14:59:54 2022 +0800
[fix](be): fix stack overflow in unhex function (#11204)
* [fix](be): fix stack overflow in unhex function
---
be/src/exprs/math_functions.cpp | 5 +++--
be/src/udf/udf.cpp| 4 ++--
be/src/udf/udf.h | 4 +++-
be/test/exprs/math_functions_test.cpp | 5 -
4 files changed, 12 insertions(+), 6 deletions(-)
diff --git a/be/src/exprs/math_functions.cpp b/be/src/exprs/math_functions.cpp
index cfda429db1..f21ebb2c62 100644
--- a/be/src/exprs/math_functions.cpp
+++ b/be/src/exprs/math_functions.cpp
@@ -381,7 +381,8 @@ StringVal MathFunctions::unhex(FunctionContext* ctx, const
StringVal& s) {
}
int result_len = s.len / 2;
-char result[result_len];
+StringVal result_string_val(ctx, result_len);
+char* result = reinterpret_cast(result_string_val.ptr);
int res_index = 0;
int s_index = 0;
while (s_index < s.len) {
@@ -426,7 +427,7 @@ StringVal MathFunctions::unhex(FunctionContext* ctx, const
StringVal& s) {
result[res_index] = c;
++res_index;
}
-return AnyValUtil::from_buffer_temp(ctx, result, result_len);
+return result_string_val;
}
StringVal MathFunctions::conv_int(FunctionContext* ctx, const BigIntVal& num,
diff --git a/be/src/udf/udf.cpp b/be/src/udf/udf.cpp
index d73fc6d691..7f72be40df 100644
--- a/be/src/udf/udf.cpp
+++ b/be/src/udf/udf.cpp
@@ -209,11 +209,11 @@ FunctionContext* FunctionContextImpl::clone(MemPool*
pool) {
namespace doris_udf {
static const int MAX_WARNINGS = 1000;
-FunctionContext* FunctionContext::create_test_context() {
+FunctionContext* FunctionContext::create_test_context(doris::MemPool* mem_pool
= nullptr) {
FunctionContext* context = new FunctionContext();
context->impl()->_debug = true;
context->impl()->_state = nullptr;
-context->impl()->_pool = new doris::FreePool(nullptr);
+context->impl()->_pool = new doris::FreePool(mem_pool);
return context;
}
diff --git a/be/src/udf/udf.h b/be/src/udf/udf.h
index 959deac8c4..c4e5583bfc 100644
--- a/be/src/udf/udf.h
+++ b/be/src/udf/udf.h
@@ -37,6 +37,7 @@ class BitmapValue;
class DecimalV2Value;
class DateTimeValue;
class CollectionValue;
+class MemPool;
} // namespace doris
namespace doris_udf {
@@ -255,7 +256,8 @@ public:
// Create a test FunctionContext object. The caller is responsible for
calling delete
// on it. This context has additional debugging validation enabled.
-static FunctionContext* create_test_context();
+// And the default value of mem_pool is nullprt.
+static FunctionContext* create_test_context(doris::MemPool* mem_pool);
~FunctionContext();
diff --git a/be/test/exprs/math_functions_test.cpp
b/be/test/exprs/math_functions_test.cpp
index 3cfdc4225a..e9231cbbe3 100644
--- a/be/test/exprs/math_functions_test.cpp
+++ b/be/test/exprs/math_functions_test.cpp
@@ -23,6 +23,7 @@
#include
#include "runtime/large_int_value.h"
+#include "runtime/mem_pool.h"
#include "testutil/function_utils.h"
#include "udf/udf_internal.h"
@@ -217,7 +218,9 @@ TEST_F(MathFunctionsTest, hex_string) {
}
TEST_F(MathFunctionsTest, unhex) {
-doris_udf::FunctionContext* context = new doris_udf::FunctionContext();
+MemPool mem_pool;
+doris_udf::FunctionContext* context =
+doris_udf::FunctionContext::create_test_context(&mem_pool);
EXPECT_EQ(StringVal::null(), MathFunctions::unhex(context,
StringVal::null()));
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11204: [fix](be): fix stack overflow in unhex function
yiguolei merged PR #11204: URL: https://github.com/apache/doris/pull/11204 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11268: [fix](planner)do not change non-null to nullable for outer join table slot
github-actions[bot] commented on PR #11268: URL: https://github.com/apache/doris/pull/11268#issuecomment-1197757169 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] liaoxin01 opened a new pull request, #11289: [feature-wip](unique-key-merge-on-write) update bitmap after compaction, DSIP-018
liaoxin01 opened a new pull request, #11289: URL: https://github.com/apache/doris/pull/11289 # Proposed changes Issue Number: close #xxx ## Problem Summary: see DISP-018: https://cwiki.apache.org/confluence/display/DORIS/DSIP-018%3A+Support+Merge-On-Write+implementation+for+UNIQUE+KEY+data+model This patch is for step 4-2 ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: add ADMIN-DIAGNOSE-TABLET
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new b469dfd518e add ADMIN-DIAGNOSE-TABLET
b469dfd518e is described below
commit b469dfd518e2cec6dc92acf91583ca78860b9d41
Author: jiafeng.zhang
AuthorDate: Thu Jul 28 15:28:43 2022 +0800
add ADMIN-DIAGNOSE-TABLET
---
.../ADMIN-DIAGNOSE-TABLET.md | 59 ++
.../ADMIN-DIAGNOSE-TABLET.md | 59 ++
sidebars.json | 1 +
3 files changed, 119 insertions(+)
diff --git
a/docs/sql-manual/sql-reference/Database-Administration-Statements/ADMIN-DIAGNOSE-TABLET.md
b/docs/sql-manual/sql-reference/Database-Administration-Statements/ADMIN-DIAGNOSE-TABLET.md
new file mode 100644
index 000..c684e429670
--- /dev/null
+++
b/docs/sql-manual/sql-reference/Database-Administration-Statements/ADMIN-DIAGNOSE-TABLET.md
@@ -0,0 +1,59 @@
+---
+{
+"title": "ADMIN DIAGNOSE TABLET",
+"language": "en"
+}
+---
+
+
+
+# ADMIN DIAGNOSE TABLET
+## description
+
+This statement is used to diagnose the specified tablet. The results will
show information about the tablet and some potential problems.
+
+grammar:
+
+ADMIN DIAGNOSE TABLET tblet_id
+
+illustrate:
+
+The lines of information in the result are as follows:
+1. TabletExist: Whether the Tablet exists
+2. TabletId:Tablet ID
+3. Database:The DB to which the Tablet
belongs and its ID
+4. Table: The Table to which Tablet
belongs and its ID
+5. Partition: The Partition to which the
Tablet belongs and its ID
+6. MaterializedIndex: The materialized view to which
the Tablet belongs and its ID
+7. Replicas(ReplicaId -> BackendId):Tablet replicas and their BE.
+8. ReplicasNum: Whether the number of replicas
is correct.
+9. ReplicaBackendStatus:Whether the BE node where the
replica is located is normal.
+10.ReplicaVersionStatus:Whether the version number of
the replica is normal.
+11.ReplicaStatus: Whether the replica status is
normal.
+12.ReplicaCompactionStatus: Whether the replica Compaction
status is normal.
+
+## example
+
+1. Diagnose tablet 10001
+
+ADMIN DIAGNOSE TABLET 10001;
+
+## keyword
+ADMIN,DIAGNOSE,TABLET
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Database-Administration-Statements/ADMIN-DIAGNOSE-TABLET.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Database-Administration-Statements/ADMIN-DIAGNOSE-TABLET.md
new file mode 100644
index 000..690adb15506
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/sql-reference/Database-Administration-Statements/ADMIN-DIAGNOSE-TABLET.md
@@ -0,0 +1,59 @@
+---
+{
+"title": "ADMIN DIAGNOSE TABLET",
+"language": "zh-CN"
+}
+---
+
+
+
+# ADMIN DIAGNOSE TABLET
+## description
+
+该语句用于诊断指定 tablet。结果中将显示这个 tablet 的信息和一些潜在的问题。
+
+语法：
+
+ADMIN DIAGNOSE TABLET tblet_id
+
+说明：
+
+结果中的各行信息如下：
+1. TabletExist: Tablet是否存在
+2. TabletId:Tablet ID
+3. Database:Tablet 所属 DB 和其 ID
+4. Table: Tablet 所属 Table 和其 ID
+5. Partition: Tablet 所属 Partition 和其 ID
+6. MaterializedIndex: Tablet 所属物化视图和其 ID
+7. Replicas(ReplicaId -> BackendId):Tablet 各副本和其所在 BE。
+8. ReplicasNum: 副本数量是否正确。
+9. ReplicaBackendStatus:副本所在 BE 节点是否正常。
+10.ReplicaVersionStatus:副本的版本号是否正常。
+11.ReplicaStatus: 副本状态是否正常。
+12.ReplicaCompactionStatus: 副本 Compaction 状态是否正常。
+
+## example
+
+1. 查看 Tablet 10001 的诊断结果
+
+ADMIN DIAGNOSE TABLET 10001;
+
+## keyword
+ADMIN,DIAGNOSE,TABLET
diff --git a/sidebars.json b/sidebars.json
index 0c7355af741..e6a8749cb30 100644
--- a/sidebars.json
+++ b/sidebars.json
@@ -646,6 +646,7 @@
"sql-manual/sql-reference/Database-Administration-Statements/ADMIN-SHOW-CONFIG",
"sql-manual/sql-reference/Database-Administration-Statements/ADMIN-SHOW-REPLICA-DISTRIBUTION",
"sql-manual/sql-reference/Database-Administration-Statements/ADMIN-SHOW-REPLICA-ST
[GitHub] [doris] yiguolei commented on a diff in pull request #11131: [Improvement] support tablet schema cache
yiguolei commented on code in PR #11131: URL: https://github.com/apache/doris/pull/11131#discussion_r931882433 ## be/src/olap/schema_change.cpp: ## @@ -1809,7 +1811,7 @@ Status SchemaChangeHandler::_do_process_alter_tablet_v2(const TAlterTabletReqV2& // delete handlers for new tablet DeleteHandler delete_handler; std::vector return_columns; -auto base_tablet_schema = base_tablet->tablet_schema(); +auto base_tablet_schema = *base_tablet->tablet_schema(); Review Comment: use copy from here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen opened a new pull request, #11290: [feature-wip](unique-key-merge-on-write) some followup of #11057
zhannngchen opened a new pull request, #11290: URL: https://github.com/apache/doris/pull/11290 # Proposed changes Issue Number: close #xxx ## Problem Summary: some followup of #11057 ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #11279: add json profile and add session context and remove olap limit
morningman commented on PR #11279: URL: https://github.com/apache/doris/pull/11279#issuecomment-1197787617 Deplicate with #10850 , fast merge -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman merged pull request #11279: add json profile and add session context and remove olap limit
morningman merged PR #11279: URL: https://github.com/apache/doris/pull/11279 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [improvement](profile) add json profile and add session context (#11279)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 97874dd125 [improvement](profile) add json profile and add session
context (#11279)
97874dd125 is described below
commit 97874dd125b34b6229cc529ba1a9b80fc8c6d7be
Author: miswujian <38979663+miswuj...@users.noreply.github.com>
AuthorDate: Thu Jul 28 15:48:00 2022 +0800
[improvement](profile) add json profile and add session context (#11279)
1. Add a new session varible "session_context"
2. support export profile in json format
---
.../org/apache/doris/analysis/SchemaTableType.java | 1 +
.../org/apache/doris/catalog/PrimitiveType.java| 3 +-
.../apache/doris/common/profile/CounterNode.java | 4 +
.../doris/common/profile/ProfileTreeNode.java | 36
.../doris/common/profile/ProfileTreePrinter.java | 39 +++-
.../apache/doris/common/util/ProfileManager.java | 17 +-
.../apache/doris/common/util/RuntimeProfile.java | 2 +-
.../apache/doris/httpv2/rest/MetaInfoAction.java | 2 +-
.../httpv2/rest/manager/QueryProfileAction.java| 212 ++---
.../java/org/apache/doris/mysql/MysqlCommand.java | 1 +
.../java/org/apache/doris/qe/SessionVariable.java | 38
.../java/org/apache/doris/qe/StmtExecutor.java | 1 +
.../doris/common/util/RuntimeProfileTest.java | 2 +-
13 files changed, 284 insertions(+), 74 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/SchemaTableType.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/SchemaTableType.java
index 127285bcb5..d4f8e68f54 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SchemaTableType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SchemaTableType.java
@@ -68,6 +68,7 @@ public enum SchemaTableType {
SCH_INVALID("NULL", "NULL", TSchemaTableType.SCH_INVALID);
private static final String dbName = "INFORMATION_SCHEMA";
private static SelectList fullSelectLists;
+
static {
fullSelectLists = new SelectList();
fullSelectLists.addItem(SelectListItem.createStarItem(null));
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/PrimitiveType.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/PrimitiveType.java
index 240ecc037a..7dd410410a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/PrimitiveType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/PrimitiveType.java
@@ -92,6 +92,7 @@ public enum PrimitiveType {
}
private static ImmutableSetMultimap
implicitCastMap;
+
static {
ImmutableSetMultimap.Builder builder =
ImmutableSetMultimap.builder();
// Nulltype
@@ -1107,7 +1108,7 @@ public enum PrimitiveType {
case DATETIMEV2: {
if (isTimeType) {
return MysqlColType.MYSQL_TYPE_TIME;
-} else {
+} else {
return MysqlColType.MYSQL_TYPE_DATETIME;
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/CounterNode.java
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/CounterNode.java
index 86f2d1a010..5934f05f05 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/profile/CounterNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/profile/CounterNode.java
@@ -27,6 +27,10 @@ public class CounterNode extends TreeNode {
counter = Pair.create(key, value);
}
+public Pair getCounter() {
+return counter;
+}
+
public String toTree(int indent) {
StringBuilder sb = new StringBuilder();
sb.append(debugString(indent));
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/ProfileTreeNode.java
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/ProfileTreeNode.java
index 45cf25d603..c6ef8b5c82 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/profile/ProfileTreeNode.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/profile/ProfileTreeNode.java
@@ -21,6 +21,8 @@ import org.apache.doris.common.TreeNode;
import com.google.common.base.Strings;
import com.google.common.collect.Lists;
+import org.json.simple.JSONArray;
+import org.json.simple.JSONObject;
import java.util.List;
@@ -155,6 +157,40 @@ public class ProfileTreeNode extends
TreeNode {
return sb.toString();
}
+public JSONObject debugStringInJson(ProfileTreePrinter.PrintLevel level,
String nodeLevel) {
+JSONObject jsonObject = new JSONObject();
+jsonObject.put("id", nodeLevel);
+JSONObject title = new JSONObject();
+if (!id.equals(ProfileTreeBuilder.UNKNOWN_ID)) {
+title.put("id", id);
+}
+title.put("name", name);
+jsonObject.put("title", title
[GitHub] [doris] github-actions[bot] commented on pull request #11181: [Regression] Add tpch plan check
github-actions[bot] commented on PR #11181: URL: https://github.com/apache/doris/pull/11181#issuecomment-1197796086 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei opened a new pull request, #11291: [fix](be): fix stack overflow in unhex function (#11204)
yiguolei opened a new pull request, #11291: URL: https://github.com/apache/doris/pull/11291 * [fix](be): fix stack overflow in unhex function # Proposed changes Issue Number: close #xxx ## Problem Summary: ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11291: [fix](be): fix stack overflow in unhex function (#11204)
yiguolei merged PR #11291: URL: https://github.com/apache/doris/pull/11291 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch dev-1.1.2 updated: [fix](be): fix stack overflow in unhex function (#11204) (#11291)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch dev-1.1.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/dev-1.1.2 by this push:
new ee83ab908e [fix](be): fix stack overflow in unhex function (#11204)
(#11291)
ee83ab908e is described below
commit ee83ab908e261305b19a254e7d57905ce85728bc
Author: yiguolei <676222...@qq.com>
AuthorDate: Thu Jul 28 15:56:49 2022 +0800
[fix](be): fix stack overflow in unhex function (#11204) (#11291)
* [fix](be): fix stack overflow in unhex function
Co-authored-by: spaces-x
---
be/src/exprs/math_functions.cpp | 5 +-
be/src/udf/udf.cpp| 4 +-
be/src/udf/udf.h | 4 +-
be/test/exprs/math_functions_test.cpp | 123 ++
4 files changed, 58 insertions(+), 78 deletions(-)
diff --git a/be/src/exprs/math_functions.cpp b/be/src/exprs/math_functions.cpp
index 13fe610971..db084fa831 100644
--- a/be/src/exprs/math_functions.cpp
+++ b/be/src/exprs/math_functions.cpp
@@ -377,7 +377,8 @@ StringVal MathFunctions::unhex(FunctionContext* ctx, const
StringVal& s) {
}
int result_len = s.len / 2;
-char result[result_len];
+StringVal result_string_val(ctx, result_len);
+char* result = reinterpret_cast(result_string_val.ptr);
int res_index = 0;
int s_index = 0;
while (s_index < s.len) {
@@ -422,7 +423,7 @@ StringVal MathFunctions::unhex(FunctionContext* ctx, const
StringVal& s) {
result[res_index] = c;
++res_index;
}
-return AnyValUtil::from_buffer_temp(ctx, result, result_len);
+return result_string_val;
}
StringVal MathFunctions::conv_int(FunctionContext* ctx, const BigIntVal& num,
diff --git a/be/src/udf/udf.cpp b/be/src/udf/udf.cpp
index 6fb8e53a0e..ea97c3848f 100644
--- a/be/src/udf/udf.cpp
+++ b/be/src/udf/udf.cpp
@@ -217,11 +217,11 @@ void FunctionContextImpl::derialize(const
PFunctionContext& pcontext) {}
namespace doris_udf {
static const int MAX_WARNINGS = 1000;
-FunctionContext* FunctionContext::create_test_context() {
+FunctionContext* FunctionContext::create_test_context(doris::MemPool* mem_pool
= nullptr) {
FunctionContext* context = new FunctionContext();
context->impl()->_debug = true;
context->impl()->_state = nullptr;
-context->impl()->_pool = new doris::FreePool(nullptr);
+context->impl()->_pool = new doris::FreePool(mem_pool);
return context;
}
diff --git a/be/src/udf/udf.h b/be/src/udf/udf.h
index 481f19a14e..5134b605e3 100644
--- a/be/src/udf/udf.h
+++ b/be/src/udf/udf.h
@@ -35,6 +35,7 @@ class BitmapValue;
class DecimalV2Value;
class DateTimeValue;
struct CollectionValue;
+class MemPool;
} // namespace doris
namespace doris_udf {
@@ -238,7 +239,8 @@ public:
// Create a test FunctionContext object. The caller is responsible for
calling delete
// on it. This context has additional debugging validation enabled.
-static FunctionContext* create_test_context();
+// And the default value of mem_pool is nullprt.
+static FunctionContext* create_test_context(doris::MemPool* mem_pool);
~FunctionContext();
diff --git a/be/test/exprs/math_functions_test.cpp
b/be/test/exprs/math_functions_test.cpp
index c9008d89ad..751f8f8fee 100644
--- a/be/test/exprs/math_functions_test.cpp
+++ b/be/test/exprs/math_functions_test.cpp
@@ -24,6 +24,8 @@
#include "exprs/anyval_util.h"
#include "exprs/expr_context.h"
+#include "runtime/large_int_value.h"
+#include "runtime/mem_pool.h"
#include "test_util/test_util.h"
#include "testutil/function_utils.h"
#include "util/logging.h"
@@ -171,100 +173,75 @@ TEST_F(MathFunctionsTest, rand) {
TEST_F(MathFunctionsTest, hex_int) {
doris_udf::FunctionContext* context = new doris_udf::FunctionContext();
-
-ASSERT_EQ(StringVal::null(),
- MathFunctions::hex_string(context, StringVal::null()));
-
-ASSERT_EQ(StringVal("7FFF"),
- MathFunctions::hex_int(context,
BigIntVal(9223372036854775807))); //BigIntVal max_value
-
+
+ASSERT_EQ(StringVal::null(), MathFunctions::hex_string(context,
StringVal::null()));
+
+ASSERT_EQ(
+StringVal("7FFF"),
+MathFunctions::hex_int(context, BigIntVal(9223372036854775807)));
//BigIntVal max_value
+
ASSERT_EQ(StringVal("FFE5853AB393E6C0"),
MathFunctions::hex_int(context, BigIntVal(-7453337203775808)));
-
-ASSERT_EQ(StringVal("0"),
- MathFunctions::hex_int(context, BigIntVal(0)));
-
-ASSERT_EQ(StringVal("C"),
- MathFunctions::hex_int(context, BigIntVal(12)));
-
-ASSERT_EQ(StringVal("90"),
- MathFunctions::hex_int(context, BigIntVal(144)));
-
-ASSERT_EQ(StringVal(""),
- MathFunctions::hex_int(context, BigIntVal(-1)));
-
-ASS
[GitHub] [doris] SaintBacchus opened a new pull request, #11292: [feature-wip][multi-catalog]Support prefetch for orc file format
SaintBacchus opened a new pull request, #11292: URL: https://github.com/apache/doris/pull/11292 # Proposed changes Issue Number: close #xxx ## Problem Summary: ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [x] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [x] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 5. Does it need to update dependencies: - [ ] Yes - [x] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11205: [feature-wip](unique-key-merge-on-write) remove AggType on unique table with MoW, enable preAggreation, DSIP-018[5/2]
github-actions[bot] commented on PR #11205: URL: https://github.com/apache/doris/pull/11205#issuecomment-1197812594 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11205: [feature-wip](unique-key-merge-on-write) remove AggType on unique table with MoW, enable preAggreation, DSIP-018[5/2]
github-actions[bot] commented on PR #11205: URL: https://github.com/apache/doris/pull/11205#issuecomment-1197812634 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11222: [feature](nereids): polish property deriver enforcer job
morrySnow commented on code in PR #11222:
URL: https://github.com/apache/doris/pull/11222#discussion_r931764521
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpec.java:
##
@@ -21,52 +21,38 @@
import org.apache.doris.nereids.memo.GroupExpression;
import org.apache.doris.nereids.trees.plans.GroupPlan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalDistribution;
-import org.apache.doris.planner.DataPartition;
import com.google.common.collect.Lists;
/**
* Spec of data distribution.
+ * GPORCA has more type in CDistributionSpec.
*/
-public class DistributionSpec {
-
-private DataPartition dataPartition;
-
-// TODO: why exist?
-public DistributionSpec() {
-}
-
-public DistributionSpec(DataPartition dataPartition) {
-this.dataPartition = dataPartition;
-}
-
+public abstract class DistributionSpec {
/**
* TODO: need read ORCA.
* Whether other `DistributionSpec` is satisfied the current
`DistributionSpec`.
*
* @param other another DistributionSpec.
*/
-public boolean meet(DistributionSpec other) {
-return false;
-}
-
-public DataPartition getDataPartition() {
-return dataPartition;
-}
-
-public void setDataPartition(DataPartition dataPartition) {
-this.dataPartition = dataPartition;
-}
+public abstract boolean meet(DistributionSpec other);
public GroupExpression addEnforcer(Group child) {
PhysicalDistribution distribution = new PhysicalDistribution(
-new DistributionSpec(dataPartition),
child.getLogicalProperties(), new GroupPlan(child));
+this,
+child.getLogicalProperties(),
Review Comment:
i doubt u need to new a LogicalProperties or just do not set logical
properties for this node. If we don't set LogicalProperties explicitly, the
node will compute a applicable LogicalProperties for itself.
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpecHash.java:
##
@@ -0,0 +1,102 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.properties;
+
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+
+/**
+ * Describe hash distribution.
+ */
+public class DistributionSpecHash extends DistributionSpec {
+
+private final List shuffledColumns;
+private final ShuffleType shuffleType;
+private PropertyInfo propertyInfo;
+
+public DistributionSpecHash(List shuffledColumns,
ShuffleType shuffleType) {
+Preconditions.checkState(!shuffledColumns.isEmpty());
+this.shuffledColumns = shuffledColumns;
+this.shuffleType = shuffleType;
+}
+
+public DistributionSpecHash(List shuffledColumns,
ShuffleType shuffleType,
+PropertyInfo propertyInfo) {
+Preconditions.checkState(!shuffledColumns.isEmpty());
+this.shuffledColumns = shuffledColumns;
+this.shuffleType = shuffleType;
+this.propertyInfo = propertyInfo;
+}
+
+public List getShuffledColumns() {
+return shuffledColumns;
+}
+
+public ShuffleType getShuffleType() {
+return shuffleType;
+}
+
+public PropertyInfo getPropertyInfo() {
+return propertyInfo;
+}
+
+@Override
+public boolean meet(DistributionSpec other) {
+if (other instanceof DistributionSpecAny) {
+return true;
+}
+
+if (!(other instanceof DistributionSpecHash)) {
+return false;
+}
+
+// DistributionSpecHash spec = (DistributionSpecHash) other;
Review Comment:
add a TODO and add @Developing annotation on class
##
fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostCalculator.java:
##
@@ -62,41 +64,73 @@ public CostEstimate visit(Plan plan, PlanContext context) {
}
@Override
-public CostEstimate visitPhysicalAggregate(PhysicalAggregate
aggregate, PlanContext context) {
+public CostEstimate visitPhysicalOlapScan(PhysicalOlapScan
[GitHub] [doris] github-actions[bot] commented on pull request #11273: [doc] Add missing ADMIN DIAGNOSE TABLET doc
github-actions[bot] commented on PR #11273: URL: https://github.com/apache/doris/pull/11273#issuecomment-1197820570 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11273: [doc] Add missing ADMIN DIAGNOSE TABLET doc
github-actions[bot] commented on PR #11273: URL: https://github.com/apache/doris/pull/11273#issuecomment-1197820513 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary opened a new issue, #11293: [Enhancement] Support alter table and column comment
stalary opened a new issue, #11293: URL: https://github.com/apache/doris/issues/11293 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description Support alter table and column comment. ### Solution Sometimes, comment needs to be modified. Comment is not associated with the data, so we should provide a lighter way of modification instead of rebuilding ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
morrySnow commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r931936217
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/EliminateAliasNode.java:
##
@@ -0,0 +1,79 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.analysis;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Eliminate the logical sub query and alias node after analyze and before
rewrite
+ * If we match the alias node and return its child node, in the execute() of
the job
+ *
+ * TODO: refactor group merge strategy to support the feature above
+ */
+public class EliminateAliasNode implements AnalysisRuleFactory {
+@Override
+public List buildRules() {
+return ImmutableList.of(
+RuleType.PROJECT_ELIMINATE_ALIAS_NODE.build(
+logicalProject().then(project ->
eliminateSubQueryAliasNode(project, project.children()))
Review Comment:
i think logicalJoin could use
```
logicalJoin(logicalSubqueryAlias(), group())
+
logicalJoin(group(), logicalSubqueryAlias())
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on a diff in pull request #11222: [feature](nereids): polish property deriver enforcer job
jackwener commented on code in PR #11222:
URL: https://github.com/apache/doris/pull/11222#discussion_r931937670
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpecAny.java:
##
@@ -18,8 +18,16 @@
package org.apache.doris.nereids.properties;
/**
- * Describe random distribution.
+ * Data can be anywhere on the segments (required only).
*/
-public class RandomDistributionDesc extends DistributionSpec {
+public class DistributionSpecAny extends DistributionSpec {
+public DistributionSpecAny() {
Review Comment:
This is learn GPORCA name.
I think its advantage is `Distribution` is put together in files list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
morrySnow commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r931938267
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSubQueryAlias.java:
##
@@ -0,0 +1,41 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.analysis;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSubQueryAlias;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * Rule to bind relation when encounter sub query and alias
+ */
+public class BindSubQueryAlias implements AnalysisRuleFactory {
+@Override
+public List buildRules() {
+return ImmutableList.of(
+RuleType.BINDING_ALIAS_SLOT.build(
+logicalSubQueryAlias().then(alias -> new
LogicalSubQueryAlias<>(alias.getAlias(), alias.child())
Review Comment:
just replace logical properties implicitly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on a diff in pull request #11222: [feature](nereids): polish property deriver enforcer job
jackwener commented on code in PR #11222:
URL: https://github.com/apache/doris/pull/11222#discussion_r931939892
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/RequestPropertyDeriver.java:
##
@@ -0,0 +1,127 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.properties;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.PlanContext;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalHashJoin;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+
+/**
+ * Used for parent property drive.
+ */
+public class RequestPropertyDeriver extends PlanVisitor {
Review Comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: add 1.1.1 release note (#21)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 3717e869104 add 1.1.1 release note (#21)
3717e869104 is described below
commit 3717e86910454cddf48ef4f896765b307d332978
Author: Luzhijing <82810928+luzhij...@users.noreply.github.com>
AuthorDate: Thu Jul 28 16:43:12 2022 +0800
add 1.1.1 release note (#21)
* Update 1.1 release note
---
docs/releasenotes/release-1.1.1.md | 61 +
.../current/releasenotes/release-1.1.1.md | 64 ++
sidebars.json | 3 +-
3 files changed, 127 insertions(+), 1 deletion(-)
diff --git a/docs/releasenotes/release-1.1.1.md
b/docs/releasenotes/release-1.1.1.md
new file mode 100644
index 000..b1bcc46612a
--- /dev/null
+++ b/docs/releasenotes/release-1.1.1.md
@@ -0,0 +1,61 @@
+---
+{
+"title": "Release 1.1.1",
+"language": "en"
+}
+---
+
+
+
+## Features
+
+### Support ODBC Sink in Vectorized Engine.
+
+This feature is enabled in non-vectorized engine but it is missed in
vectorized engine in 1.1. So that we add back this feature in 1.1.1.
+
+### Simple Memtracker for Vectorized Engine.
+
+There is no memtracker in BE for vectorized engine in 1.1, so that the memory
is out of control and cause OOM. In 1.1.1, a simple memtracker is added to BE
and could control the memory and cancel the query when memory exceeded.
+
+## Improvements
+
+### Cache decompressed data in page cache.
+
+Some data is compressed using bitshuffle and it costs a lot of time to
decompress it during query. In 1.1.1, doris will decompress the data that
encoded by bitshuffle to accelerate query and we find it could reduce 30%
latency for some query in ssb-flat.
+
+## Bug Fix
+
+### Fix the problem that could not do rolling upgrade from 1.0.(Serious)
+
+This issue was introduced in version 1.1 and may cause BE core when upgrade BE
but not upgrade FE.
+
+If you encounter this problem, you can try to fix it with
[#10833](https://github.com/apache/doris/pull/10833).
+
+### Fix the problem that some query not fall back to non-vectorized engine,
and BE will core.
+
+Currently, vectorized engine could not deal with all sql queries and some
queries (like left outer join) will use non-vectorized engine to run. But there
are some cases not covered in 1.1. And it will cause be crash.
+
+### Compaction not work correctly and cause -235 Error.
+
+One rowset multi segments in uniq key compaction, segments rows will be merged
in generic_iterator but merged_rows not increased. Compaction will failed in
check_correctness, and make a tablet with too much versions which lead to -235
load error.
+
+### Some segment fault cases during query.
+
+[#10961](https://github.com/apache/doris/pull/10961)
[#10954](https://github.com/apache/doris/pull/10954)
[#10962](https://github.com/apache/doris/pull/10962)
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/releasenotes/release-1.1.1.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/releasenotes/release-1.1.1.md
new file mode 100644
index 000..d799fac32c1
--- /dev/null
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/releasenotes/release-1.1.1.md
@@ -0,0 +1,64 @@
+---
+{
+"title": "Release 1.1.0",
+"language": "zh-CN"
+}
+---
+
+
+
+## 新增功能
+
+### 向量化执行引擎支持 ODBC Sink。
+
+在 1.1.0 版本的向量化执行引擎中 ODBC Sink 是不支持的，而这一功能在之前版本的行存引擎是支持的，因此在 1.1.1
版本中我们重新完善了这一功能。
+
+### 增加简易版 MemTracker
+
+MemTracker 是一个用于分析内存使用情况的统计工具，在 1.1.0 版本的向量化执行引擎中，由于 BE 侧没有
MemTracker，可能出现因内存失控导致的 OOM 问题。在 1.1.1 版本中，BE 侧增加了一个简易版
MemTracker，可以帮助控制内存，并在内存超出时取消查询。
+
+完整版 MemTracker 将在 1.1.2 版本中正式发布。
+
+
+## 改进
+
+### 支持在 Page Cache 中缓存解压后数据。
+
+在 Page Cache 中有些数据是用 bitshuffle 编码方式压缩的，在查询过程中需要花费大量的时间来解压。在 1.1.1 版本中，Doris
将缓存解压由 bitshuffle 编码的数据以加速查询，我们发现在 ssb-flat 的一些查询中，可以减少 30% 的延时。
+
+## Bug 修复
+
+### 修复无法从 1.0 版本进行滚动升级的问题。
+
+这个问题是在 1.1.0 版本中出现的，当升级 BE 而不升级 FE 时，可能会导致 BE Core。
+
+如果你遇到这个问题，你可以尝试用 [#10833](https://github.com/apache/doris/pull/10833) 来修复它。
+
+### 修复某些查询不能回退到非向量化引擎的问题，并导致 BE Core。
+
+目前，向量化执行引擎不能处理所有的 SQL 查询，一些查询（如 left outer join）将使用非向量化引擎来运行。但部分场景在 1.1.0
版本中未被覆盖到，这可能导致 BE 挂掉。
+
+### Compaction 不能正常工作，导致 -235 错误。
+
+在 Unique Key 模型中，当一个 Rowset 有多个 Segment 时，在做 Compaction
过程中由于没有正确的统计行数，会导致Compaction 失败并且产生 Tablet 版本过多而导致的 -235 错误。
+
+### 在查询过程中出现了一些分段故障。
+
+[#10961](https://github.com/apache/doris/pull/10961)
[#10954](https://github.com/apache/doris/pull/10954)
[#10962](https://github.com/apache/doris/pull/10962)
diff --git a/sidebars.json b/sidebars.json
index e6a8749cb30..53771fe5e9a 100644
--- a/sidebars.json
+++ b/sidebars.json
@@ -916,7 +916,8 @@
"label": "Release notes",
"items": [
"releasenotes/release-1.1.0"
+"releas
[GitHub] [doris] stalary commented on issue #11293: [Enhancement] Support alter table and column comment
stalary commented on issue #11293: URL: https://github.com/apache/doris/issues/11293#issuecomment-1197848420 I find this demand already implemented, but without doc, so I will add doc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] englefly commented on a diff in pull request #11209: [feature] (Nereids) support limit clause
englefly commented on code in PR #11209:
URL: https://github.com/apache/doris/pull/11209#discussion_r931959907
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalLimit.java:
##
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.IntegerLiteral;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * Logical limit plan
+ * eg: select * from table limit 10
+ * limit: 10
+ *
+ * eg: select * from table order by a limit 100, 10
+ * limit: 10
+ * offset 100
+ */
+public class LogicalLimit extends
LogicalUnary {
+private long limit = Long.MAX_VALUE;
+private long offset = 0L;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] stalary opened a new pull request, #11294: [Doc] Add alter table comment doc
stalary opened a new pull request, #11294: URL: https://github.com/apache/doris/pull/11294 # Proposed changes Issue Number: close https://github.com/apache/doris/issues/11293 ## Problem Summary: ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [feature-wip](unique-key-merge-on-write) remove AggType on unique table with MoW, enable preAggreation, DSIP-018[5/2] (#11205)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 70c7e3d7aa [feature-wip](unique-key-merge-on-write) remove AggType on
unique table with MoW, enable preAggreation, DSIP-018[5/2] (#11205)
70c7e3d7aa is described below
commit 70c7e3d7aa30cfdbddbc6ba1725c22f270343c9b
Author: zhannngchen <48427519+zhannngc...@users.noreply.github.com>
AuthorDate: Thu Jul 28 17:03:05 2022 +0800
[feature-wip](unique-key-merge-on-write) remove AggType on unique table
with MoW, enable preAggreation, DSIP-018[5/2] (#11205)
remove AggType on unique table with MoW, enable preAggreation
---
be/src/olap/delta_writer.cpp | 5 +-
be/src/olap/memtable.cpp | 87 +++---
be/src/olap/memtable.h | 14 ++--
.../org/apache/doris/analysis/CreateTableStmt.java | 19 -
.../doris/datasource/InternalDataSource.java | 5 ++
.../doris/planner/MaterializedViewSelector.java| 22 --
.../org/apache/doris/planner/OlapScanNode.java | 8 +-
.../apache/doris/analysis/CreateTableStmtTest.java | 46
.../planner/MaterializedViewSelectorTest.java | 8 +-
9 files changed, 163 insertions(+), 51 deletions(-)
diff --git a/be/src/olap/delta_writer.cpp b/be/src/olap/delta_writer.cpp
index 04b7327f25..e50a8567b4 100644
--- a/be/src/olap/delta_writer.cpp
+++ b/be/src/olap/delta_writer.cpp
@@ -278,9 +278,8 @@ Status DeltaWriter::wait_flush() {
}
void DeltaWriter::_reset_mem_table() {
-_mem_table.reset(new MemTable(_tablet->tablet_id(), _schema.get(),
_tablet_schema.get(),
- _req.slots, _req.tuple_desc,
_tablet->keys_type(),
- _rowset_writer.get(), _is_vec));
+_mem_table.reset(new MemTable(_tablet, _schema.get(),
_tablet_schema.get(), _req.slots,
+ _req.tuple_desc, _rowset_writer.get(),
_is_vec));
}
Status DeltaWriter::close() {
diff --git a/be/src/olap/memtable.cpp b/be/src/olap/memtable.cpp
index 46607f6a6c..a1ade071d0 100644
--- a/be/src/olap/memtable.cpp
+++ b/be/src/olap/memtable.cpp
@@ -29,16 +29,15 @@
namespace doris {
-MemTable::MemTable(int64_t tablet_id, Schema* schema, const TabletSchema*
tablet_schema,
+MemTable::MemTable(TabletSharedPtr tablet, Schema* schema, const TabletSchema*
tablet_schema,
const std::vector* slot_descs,
TupleDescriptor* tuple_desc,
- KeysType keys_type, RowsetWriter* rowset_writer, bool
support_vec)
-: _tablet_id(tablet_id),
+ RowsetWriter* rowset_writer, bool support_vec)
+: _tablet(std::move(tablet)),
_schema(schema),
_tablet_schema(tablet_schema),
_slot_descs(slot_descs),
- _keys_type(keys_type),
_mem_tracker(std::make_unique(
- fmt::format("MemTable:tabletId={}",
std::to_string(tablet_id,
+ fmt::format("MemTable:tabletId={}",
std::to_string(tablet_id(),
_buffer_mem_pool(new MemPool(_mem_tracker.get())),
_table_mem_pool(new MemPool(_mem_tracker.get())),
_schema_size(_schema->schema_size()),
@@ -52,18 +51,19 @@ MemTable::MemTable(int64_t tablet_id, Schema* schema, const
TabletSchema* tablet
_skip_list = nullptr;
_vec_row_comparator = std::make_shared(_schema);
// TODO: Support ZOrderComparator in the future
-_vec_skip_list = std::make_unique(
-_vec_row_comparator.get(), _table_mem_pool.get(), _keys_type
== KeysType::DUP_KEYS);
+_vec_skip_list =
+std::make_unique(_vec_row_comparator.get(),
_table_mem_pool.get(),
+ keys_type() == KeysType::DUP_KEYS);
_init_columns_offset_by_slot_descs(slot_descs, tuple_desc);
} else {
_vec_skip_list = nullptr;
-if (_keys_type == KeysType::DUP_KEYS) {
+if (keys_type() == KeysType::DUP_KEYS) {
_insert_fn = &MemTable::_insert_dup;
} else {
_insert_fn = &MemTable::_insert_agg;
}
-if (_tablet_schema->has_sequence_col()) {
-_aggregate_two_row_fn =
&MemTable::_aggregate_two_row_with_sequence;
+if (keys_type() == KeysType::UNIQUE_KEYS &&
_tablet->enable_unique_key_merge_on_write()) {
+_aggregate_two_row_fn = &MemTable::_replace_row;
} else {
_aggregate_two_row_fn = &MemTable::_aggregate_two_row;
}
@@ -74,7 +74,7 @@ MemTable::MemTable(int64_t tablet_id, Schema* schema, const
TabletSchema* tablet
_row_comparator = std::make_shared(_schema);
}
_skip_list = std::make_unique(_row_comparator.get(),
_table_mem_pool.get(),
-
[GitHub] [doris] dataroaring merged pull request #11205: [feature-wip](unique-key-merge-on-write) remove AggType on unique table with MoW, enable preAggreation, DSIP-018[5/2]
dataroaring merged PR #11205: URL: https://github.com/apache/doris/pull/11205 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-flink-connector] link3280 opened a new pull request, #51: [docs] Fix broken link for doris connector docs
link3280 opened a new pull request, #51: URL: https://github.com/apache/doris-flink-connector/pull/51 # Proposed changes The link to the website is broken. This PR fixes it. ## Problem Summary: The link to the website is broken. This PR fixes it. ## Checklist(Required) 1. Does it affect the original behavior: ni 2. Has unit tests been added: ni 3. Has document been added or modified: no 4. Does it need to update dependencies: no 5. Are there any changes that cannot be rolled back: no ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11278: [Vectorized] Support both distinct and order by of group_concat
github-actions[bot] commented on PR #11278: URL: https://github.com/apache/doris/pull/11278#issuecomment-1197872169 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11278: [Vectorized] Support both distinct and order by of group_concat
github-actions[bot] commented on PR #11278: URL: https://github.com/apache/doris/pull/11278#issuecomment-1197872120 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
morrySnow commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r931973043
##
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##
@@ -190,11 +194,39 @@ public LogicalPlan
visitRegularQuerySpecification(RegularQuerySpecificationConte
/**
* Create an aliased table reference. This is typically used in FROM
clauses.
*/
+private LogicalPlan applyAlias(TableAliasContext ctx, LogicalPlan plan) {
+if (null != ctx.strictIdentifier()) {
+String alias = ctx.strictIdentifier().getText();
+if (null != ctx.identifierList()) {
+throw new IllegalStateException("Do not implemented");
+// List colName =
visitIdentifierSeq(ctx.identifierList().identifierSeq());
+// TODO: multi-colName
+} else {
+return new LogicalSubQueryAlias<>(alias, plan);
+}
+}
+return plan;
+}
+
@Override
public LogicalPlan visitTableName(TableNameContext ctx) {
List tableId =
visitMultipartIdentifier(ctx.multipartIdentifier());
-// TODO: sample and time travel, alias, sub query
-return new UnboundRelation(tableId);
+return applyAlias(ctx.tableAlias(), new UnboundRelation(tableId));
+}
+
+@Override
+public LogicalPlan visitAliasedQuery(AliasedQueryContext ctx) {
+String alias = "__auto_generated_name__";
+LogicalPlan query = visitQuery(ctx.query());
+if (null != ctx.tableAlias().strictIdentifier()) {
Review Comment:
strictIdentifier cannot be null
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/EliminateAliasNode.java:
##
@@ -0,0 +1,91 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.analysis;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Eliminate the logical sub query and alias node after analyze and before
rewrite
+ * If we match the alias node and return its child node, in the execute() of
the job
+ *
+ * TODO: refactor group merge strategy to support the feature above
+ */
+public class EliminateAliasNode implements AnalysisRuleFactory {
+@Override
+public List buildRules() {
+return ImmutableList.of(
+RuleType.PROJECT_ELIMINATE_ALIAS_NODE.build(
+logicalProject(logicalSubQueryAlias())
+.then(project ->
eliminateSubQueryAliasNode(project, project.children()))
+),
+RuleType.FILTER_ELIMINATE_ALIAS_NODE.build(
+logicalFilter(logicalSubQueryAlias())
+.then(filter ->
eliminateSubQueryAliasNode(filter, filter.children()))
+),
+RuleType.AGGREGATE_ELIMINATE_ALIAS_NODE.build(
+logicalAggregate(logicalSubQueryAlias())
+.then(agg -> eliminateSubQueryAliasNode(agg,
agg.children()))
+),
+RuleType.JOIN_ELIMINATE_ALIAS_NODE.build(
+logicalJoin().then(join ->
joinEliminateSubQueryAliasNode(join, join.children()))
+)
+);
+}
+
+private LogicalPlan eliminateSubQueryAliasNode(LogicalPlan node,
List aliasNodes) {
+List nodes = aliasNodes.stream()
+.map(this::getPlan)
+.collect(Collectors.toList());
+return (LogicalPlan) node.withChildren(nodes);
+}
+
+private LogicalPlan joinEliminateSubQueryAliasNode(LogicalPlan node,
List aliasNode) {
+List nodes = aliasNode.stream()
+.map(child -> {
+if (checkIsSubQueryAliasNode(child)) {
+retur
[GitHub] [doris] 924060929 merged pull request #11210: [enhancement](Nereids)enable explain query for nereids planner
924060929 merged PR #11210: URL: https://github.com/apache/doris/pull/11210 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](Nereids)enable explain query for nereids planner (#11210)
This is an automated email from the ASF dual-hosted git repository.
huajianlan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 951320256b [enhancement](Nereids)enable explain query for nereids
planner (#11210)
951320256b is described below
commit 951320256b418c12e6c4de880ed4f1d70c2e1e30
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Thu Jul 28 17:23:17 2022 +0800
[enhancement](Nereids)enable explain query for nereids planner (#11210)
Support explain syntax. And generate explain tree string by nereids'
planner.
---
.../antlr4/org/apache/doris/nereids/DorisLexer.g4 | 2 +
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 4 +
.../org/apache/doris/nereids/NereidsPlanner.java | 5 ++
.../glue/translator/PhysicalPlanTranslator.java| 3 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 16
.../apache/doris/nereids/parser/NereidsParser.java | 28 --
.../org/apache/doris/nereids/pattern/Pattern.java | 7 --
.../doris/nereids/trees/AbstractTreeNode.java | 8 +-
.../org/apache/doris/nereids/trees/TreeNode.java | 6 --
.../doris/nereids/trees/plans/AbstractPlan.java| 6 ++
.../org/apache/doris/nereids/trees/plans/Plan.java | 3 +
.../nereids/trees/plans/commands/Command.java | 100 +
.../trees/plans/commands/ExplainCommand.java | 52 +++
.../doris/nereids/parser/NereidsParserTest.java| 58
14 files changed, 269 insertions(+), 29 deletions(-)
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
index 21ff03c484..c7bb9083e9 100644
--- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
+++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
@@ -191,6 +191,7 @@ FUNCTION: 'FUNCTION';
FUNCTIONS: 'FUNCTIONS';
GLOBAL: 'GLOBAL';
GRANT: 'GRANT';
+GRAPH: 'GRAPH';
GROUP: 'GROUP';
GROUPING: 'GROUPING';
HAVING: 'HAVING';
@@ -358,6 +359,7 @@ USE: 'USE';
USER: 'USER';
USING: 'USING';
VALUES: 'VALUES';
+VERBOSE: 'VERBOSE';
VERSION: 'VERSION';
VIEW: 'VIEW';
VIEWS: 'VIEWS';
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
index 64fa6c8752..6fab3a9b0c 100644
--- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
+++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
@@ -50,6 +50,8 @@ singleStatement
statement
: query
#statementDefault
+| (EXPLAIN | DESC | DESCRIBE) level=(VERBOSE | GRAPH)?
+query #explain
;
// -Query-
@@ -626,6 +628,7 @@ nonReserved
| FUNCTIONS
| GLOBAL
| GRANT
+| GRAPH
| GROUP
| GROUPING
| HAVING
@@ -779,6 +782,7 @@ nonReserved
| USE
| USER
| VALUES
+| VERBOSE
| VERSION
| VIEW
| VIEWS
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
index 625ec84097..3ccb0615d1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
@@ -177,4 +177,9 @@ public class NereidsPlanner extends Planner {
public DescriptorTable getDescTable() {
return descTable;
}
+
+@Override
+public void appendTupleInfo(StringBuilder str) {
+str.append(descTable.getExplainString());
+}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index b55ff0277e..139cf360e3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -117,7 +117,7 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor physicalProject = (PhysicalProject)
physicalPlan;
List outputExprs = physicalProject.getProjects().stream()
-.map(e -> ExpressionTranslator.translate((Expression) e,
context))
+.map(e -> ExpressionTranslator.translate(e, context))
.collect(Collectors.toList());
rootFragment.setOutputExprs(outputExprs);
} else {
@@ -329,7 +329,6 @@ public class PhysicalPlanTranslator extends
DefaultPlanVisitor {
/*
* Plan parsing
[GitHub] [doris] sohardforaname commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
sohardforaname commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r931994568
##
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##
@@ -190,11 +194,39 @@ public LogicalPlan
visitRegularQuerySpecification(RegularQuerySpecificationConte
/**
* Create an aliased table reference. This is typically used in FROM
clauses.
*/
+private LogicalPlan applyAlias(TableAliasContext ctx, LogicalPlan plan) {
+if (null != ctx.strictIdentifier()) {
+String alias = ctx.strictIdentifier().getText();
+if (null != ctx.identifierList()) {
+throw new IllegalStateException("Do not implemented");
+// List colName =
visitIdentifierSeq(ctx.identifierList().identifierSeq());
+// TODO: multi-colName
+} else {
+return new LogicalSubQueryAlias<>(alias, plan);
+}
+}
+return plan;
+}
+
@Override
public LogicalPlan visitTableName(TableNameContext ctx) {
List tableId =
visitMultipartIdentifier(ctx.multipartIdentifier());
-// TODO: sample and time travel, alias, sub query
-return new UnboundRelation(tableId);
+return applyAlias(ctx.tableAlias(), new UnboundRelation(tableId));
+}
+
+@Override
+public LogicalPlan visitAliasedQuery(AliasedQueryContext ctx) {
+String alias = "__auto_generated_name__";
+LogicalPlan query = visitQuery(ctx.query());
+if (null != ctx.tableAlias().strictIdentifier()) {
Review Comment:
Yes, I got an error at running query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sohardforaname commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
sohardforaname commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r931995398
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/EliminateAliasNode.java:
##
@@ -0,0 +1,91 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.analysis;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.stream.Collectors;
+
+/**
+ * Eliminate the logical sub query and alias node after analyze and before
rewrite
+ * If we match the alias node and return its child node, in the execute() of
the job
+ *
+ * TODO: refactor group merge strategy to support the feature above
+ */
+public class EliminateAliasNode implements AnalysisRuleFactory {
+@Override
+public List buildRules() {
+return ImmutableList.of(
+RuleType.PROJECT_ELIMINATE_ALIAS_NODE.build(
+logicalProject(logicalSubQueryAlias())
+.then(project ->
eliminateSubQueryAliasNode(project, project.children()))
+),
+RuleType.FILTER_ELIMINATE_ALIAS_NODE.build(
+logicalFilter(logicalSubQueryAlias())
+.then(filter ->
eliminateSubQueryAliasNode(filter, filter.children()))
+),
+RuleType.AGGREGATE_ELIMINATE_ALIAS_NODE.build(
+logicalAggregate(logicalSubQueryAlias())
+.then(agg -> eliminateSubQueryAliasNode(agg,
agg.children()))
+),
+RuleType.JOIN_ELIMINATE_ALIAS_NODE.build(
+logicalJoin().then(join ->
joinEliminateSubQueryAliasNode(join, join.children()))
+)
+);
+}
+
+private LogicalPlan eliminateSubQueryAliasNode(LogicalPlan node,
List aliasNodes) {
+List nodes = aliasNodes.stream()
+.map(this::getPlan)
+.collect(Collectors.toList());
+return (LogicalPlan) node.withChildren(nodes);
+}
+
+private LogicalPlan joinEliminateSubQueryAliasNode(LogicalPlan node,
List aliasNode) {
+List nodes = aliasNode.stream()
+.map(child -> {
+if (checkIsSubQueryAliasNode(child)) {
+return ((GroupPlan) child).getGroup()
+.getLogicalExpression()
+.child(0)
+.getLogicalExpression()
+.getPlan();
+}
+return child;
+})
+.collect(Collectors.toList());
+return (LogicalPlan) node.withChildren(nodes);
+}
+
+private boolean checkIsSubQueryAliasNode(Plan node) {
Review Comment:
Yes, really Ok hhh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #11295: [community](*): remove type in checklist.
jackwener opened a new pull request, #11295: URL: https://github.com/apache/doris/pull/11295 # Proposed changes Issue Number: close #xxx ## Problem Summary: Remove the PR type in PR_TEMPLATE ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on pull request #11186: Add FileCache for RemoteFile
jackwener commented on PR #11186: URL: https://github.com/apache/doris/pull/11186#issuecomment-1197906620 Please add header of title according to the Requirement https://doris.apache.org/zh-CN/community/how-to-contribute/commit-format-specification -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Gabriel39 opened a new pull request, #11296: [refactor] (datetimev2) refactor FEFunctions for datev2/datetimev2
Gabriel39 opened a new pull request, #11296: URL: https://github.com/apache/doris/pull/11296 # Proposed changes Issue Number: close #xxx ## Problem Summary: ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sohardforaname commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
sohardforaname commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r932010275
##
fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java:
##
@@ -168,6 +169,11 @@ public boolean equals(Object o) {
return false;
}
GroupExpression that = (GroupExpression) o;
+// if the plan is LogicalOlapScan, this == that should be true,
+// because equals() can not divide UnboundRelation and the plan above.
+if (plan instanceof LogicalOlapScan) {
Review Comment:
fixed
##
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##
@@ -190,11 +194,39 @@ public LogicalPlan
visitRegularQuerySpecification(RegularQuerySpecificationConte
/**
* Create an aliased table reference. This is typically used in FROM
clauses.
*/
+private LogicalPlan applyAlias(TableAliasContext ctx, LogicalPlan plan) {
+if (null != ctx.strictIdentifier()) {
+String alias = ctx.strictIdentifier().getText();
+if (null != ctx.identifierList()) {
+throw new IllegalStateException("Do not implemented");
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Wilson-BT opened a new issue, #11297: [Bug] BE memery Problem
Wilson-BT opened a new issue, #11297: URL: https://github.com/apache/doris/issues/11297 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.1.0 ### What's Wrong? when i exectue insert into yyy select from xxx，be memory will increase quickly and never release.  in the picture, the memory is used about 53GB+, but when i look at the web metric, just use 38G on every BE.  ### What You Expected? i think the unused memory on BE should release quickly, and the web mem_tracker should be precise ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 closed pull request #11203: [enhancement](nereids) Don't normalize expressions when pushing down predicates
adonis0147 closed pull request #11203: [enhancement](nereids) Don't normalize expressions when pushing down predicates URL: https://github.com/apache/doris/pull/11203 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sohardforaname commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
sohardforaname commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r932011128
##
fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java:
##
@@ -168,6 +169,11 @@ public boolean equals(Object o) {
return false;
}
GroupExpression that = (GroupExpression) o;
+// if the plan is LogicalOlapScan, this == that should be true,
+// because equals() can not divide UnboundRelation and the plan above.
Review Comment:
comment rewritten
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] freshsun commented on issue #6680: [Bug] 当并行执行sql语句BE占用内存达到mem_limit上限值时，BE进程异常退出
freshsun commented on issue #6680: URL: https://github.com/apache/doris/issues/6680#issuecomment-1197919162 +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #11298: [Improvement][chore] remove redundance fpic on thirdpatry build
BiteThet opened a new pull request, #11298: URL: https://github.com/apache/doris/pull/11298 # Proposed changes Doris will static link all thirdparty, `-fPIC` only work on dynamic link, so we can remove those. ## Problem Summary: ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] adonis0147 opened a new pull request, #11299: [enhancement](nereids) Don't normalize expressions when pushing down predicates
adonis0147 opened a new pull request, #11299: URL: https://github.com/apache/doris/pull/11299 # Proposed changes ~~Issue Number: close #xxx~~ ## Problem Summary: Rules for normalizing expressions should be applied once before do some extra expression transforms. ## Checklist(Required) 1. Type of your changes: - [x] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 5. Does it need to update dependencies: - [ ] Yes - [x] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhengshiJ opened a new pull request, #11300: [feature](Nereids) Add subquery analyze
zhengshiJ opened a new pull request, #11300: URL: https://github.com/apache/doris/pull/11300 # Proposed changes Issue Number: no ## Problem Summary: Increase the parsing of subquery. Add LogicalApply and LogicalCorrelatedJoin. LogicalApply: Apply Node for subquery. Use this node to display the subquery in the relational algebra tree. refer to "Orthogonal Optimization of Subqueries and Aggregation" LogicalCorrelatedJoin: A relational algebra node with join type converted from apply node to subquery. ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yangzhg opened a new pull request, #11301: [enhancement] Improve limit query performance on olapScannode
yangzhg opened a new pull request, #11301: URL: https://github.com/apache/doris/pull/11301 # Proposed changes Optimize the query logic with limit, for the case where the limit value is relatively small, reduce the parallelism of the scanner, reduce unnecessary resource consumption, and increase the number of similar queries that the system can carry at the same time, and increase the query speed by more than 60% before:  after:  ## Checklist(Required) 1. Type of your changes: - [x] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 5. Does it need to update dependencies: - [ ] Yes - [x] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new issue, #11302: [Bug][Function] core dump on sum(distinct)
BiteThet opened a new issue, #11302: URL: https://github.com/apache/doris/issues/11302 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version master ### What's Wrong? ```sql select sum(distinct k1), sum(distinct k2), sum(distinct k3), sum(distinct cast(k4 as largeint)), sum(distinct k5), sum(distinct k8), sum(distinct k9) from test; ``` ```cpp *** Query id: 30ee93e79fdf457f-a53e29e9ca7bac2a *** *** Aborted at 1659002173 (unix time) try "date -d @1659002173" if you are using GNU date *** *** Current BE git commitID: cf7960e13a *** *** SIGSEGV unkown detail explain (@0x0) received by PID 93226 (TID 0x7fd2ff985700) from PID 0; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/disk2/pxl/dev/baidu/bdg/doris/core/be/src/common/signal_handler.h:420 1# 0x00318AE32920 in /lib64/libc.so.6 2# doris::vectorized::AggregateFunctionSumData >::get() const at /home/disk2/pxl/dev/baidu/bdg/doris/core/be/src/vec/aggregate_functions/aggregate_function_sum.h:43 3# doris::vectorized::AggregateFunctionSum, doris::vectorized::Decimal<__int128>, doris::vectorized::AggregateFunctionSumData > >::insert_result_into(char const*, doris::vectorized::IColumn&) const at /home/disk2/pxl/dev/baidu/bdg/doris/core/be/src/vec/aggregate_functions/aggregate_function_sum.h:101 4# doris::vectorized::AggregateFunctionDistinct >::insert_result_into(char const*, doris::vectorized::IColumn&) const at /home/disk2/pxl/dev/baidu/bdg/doris/core/be/src/vec/aggregate_functions/aggregate_function_distinct.h:210 5# doris::vectorized::AggregateFunctionNullBase >::insert_result_into(char const*, doris::vectorized::IColumn&) const at /home/disk2/pxl/dev/baidu/bdg/doris/core/be/src/vec/aggregate_functions/aggregate_function_null.h:167 6# doris::vectorized::AggFnEvaluator::insert_result_info(char*, doris::vectorized::IColumn*) at /home/disk2/pxl/dev/baidu/bdg/doris/core/be/src/vec/exprs/vectorized_agg_fn.cpp:161 7# doris::vectorized::AggregationNode::_get_without_key_result(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/disk2/pxl/dev/baidu/bdg/doris/core/be/src/vec/exec/vaggregation_node.cpp:522 8# doris::Status std::__invoke_impl(std::__invoke_memfun_deref, doris::Status (doris::vectorized::AggregationNode::*&)(doris::RuntimeState*, doris::vectorized::Block*, bool*), doris::vectorized::AggregationNode*&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) at /home/disk2/pxl/dev/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74 9# std::enable_if, doris::Status>::type std::__invoke_r(doris::Status (doris::vectorized::AggregationNode::*&)(doris::RuntimeState*, doris::vectorized::Block*, bool*), doris::vectorized::AggregationNode*&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) at /home/disk2/pxl/dev/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:114 10# doris::Status std::_Bind_result, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>::__call(std::tuple&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul>) at /home/disk2/pxl/dev/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:570 11# doris::Status std::_Bind_result, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>::operator()(doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) at /home/disk2/pxl/dev/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/functional:629 12# doris::Status std::__invoke_impl, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*, doris::vectorized::Block*, bool*>(std::__invoke_other, std::_Bind_result, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) at /home/disk2/pxl/dev/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61 13# std::enable_if, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*, doris::vectorized::Block*, bool*>, doris::Status>::type std::__invoke_r, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*, doris::vectorized::Block*, bool*>(std::_Bind_result, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*&&, doris::v ectorized::Block*&&, bool*&&) at /home/disk2/pxl/d
[GitHub] [doris] liaoxin01 opened a new pull request, #11303: [feature-wip](unique-key-merge-on-write) fix IndexedColumnReader and add primary key index memory tracker, DSIP-018
liaoxin01 opened a new pull request, #11303: URL: https://github.com/apache/doris/pull/11303 # Proposed changes Issue Number: close #xxx ## Problem Summary: see DISP-018: https://cwiki.apache.org/confluence/display/DORIS/DSIP-018%3A+Support+Merge-On-Write+implementation+for+UNIQUE+KEY+data+model, fix IndexedColumnReader return not found in some case. Add primay key index memory tracker. ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-website] branch master updated: mod get starting
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new d2c749c13b2 mod get starting
d2c749c13b2 is described below
commit d2c749c13b26e5d10bb8ac142f9a8161005ff4e5
Author: jiafeng.zhang
AuthorDate: Thu Jul 28 18:29:12 2022 +0800
mod get starting
---
docs/get-starting/get-starting.md | 40 +++---
.../ADMIN-DIAGNOSE-TABLET.md | 2 +-
.../current/get-starting/get-starting.md | 150 +++--
.../ADMIN-DIAGNOSE-TABLET.md | 2 +-
4 files changed, 105 insertions(+), 89 deletions(-)
diff --git a/docs/get-starting/get-starting.md
b/docs/get-starting/get-starting.md
index d7c209faea6..1cdcfa300c7 100644
--- a/docs/get-starting/get-starting.md
+++ b/docs/get-starting/get-starting.md
@@ -40,6 +40,9 @@ under the License.
**Before creating, please prepare the compiled FE/BE file, this tutorial will
not repeat the compilation process.**
+
+**This tutorial is a mixed distribution tutorial of single node, 1FE and 1BE,
only one node is needed, which is convenient for quickly experiencing Doris.**
+
1. Set the maximum number of open file handles in the system
```shell
@@ -53,9 +56,10 @@ under the License.
2. Download binary package / self-compile FE / BE files
```shell
- wget https://dist.apache.org/repos/dist/release/incubator/doris/version to
deploy
+ # Select the version you want to deploy from the following address
+ https://archive.apache.org/dist/doris
# For example the following link
- wget
https://dist.apache.org/repos/dist/release/incubator/doris/1.0/1.0.0-incubating/apache-doris-1.0.0-incubating-bin.tar.gz
+ wget
https://archive.apache.org/dist/doris/1.1/1.1.0-rc05/apache-doris-1.1.0-bin-x86-jdk8.tar.gz
```
3. Extract the tar.gz file
@@ -63,7 +67,7 @@ under the License.
```shell
tar -zxvf Downloaded binary archive
# example
- tar -zxvf apache-doris-1.0.0-incubating-bin.tar.gz
+ tar -zxvf apache-doris-1.1.0-bin-x86-jdk8.tar.gz
```
4. Migrate the decompressed program files to the specified directory and
rename them
@@ -72,7 +76,7 @@ under the License.
mv [unzipped root directory] [Target path]
cd [Target path]
# example
- mv apache-doris-1.0.0-incubating-bin /opt/doris
+ mv apache-doris-1.1.0-bin-x86-jdk8 /opt/doris
cd /opt/doris
```
@@ -82,7 +86,9 @@ under the License.
# Configure FE-Config
vi fe/conf/fe.conf
# Uncomment priority_networks and modify parameters
- priority_networks = 127.0.0.0/24
+ # For example, if the IP address of the current node is 10.10.2.21, you
need to change it to 10.10.2.0/24 and fill in
+ # What needs to be filled in here is the IP subnet address, not the IP
address
+ priority_networks = 10.10.2.0/24
# save and exit
```
@@ -92,7 +98,9 @@ under the License.
# Configure BE-Config
vi be/conf/be.conf
# Uncomment priority_networks and modify parameters
- priority_networks = 127.0.0.0/24
+ # For example, if the IP address of the current node is 10.10.2.21, you
need to change it to 10.10.2.0/24 and fill in
+ # What needs to be filled in here is the IP subnet address, not the IP
address
+ priority_networks = 10.10.2.0/24
# save and exit
```
@@ -117,7 +125,7 @@ under the License.
Check whether the FE startup is successful
> 1. Check whether the startup is successful, and whether there is a PaloFe
process under the JPS command
- > 2. After the FE process is started, the metadata will be loaded first.
Depending on the role of the FE, you will see transfer from UNKNOWN to
MASTER/FOLLOWER/OBSERVER in the log. Finally, you will see the thrift server
started log, and you can connect to FE through the mysql client, which means FE
started successfully.
+ > 2. After the FE process is started, the metadata will be loaded first.
Depending on the role of the FE, you will see transfer from UNKNOWN to
MASTER/FOLLOWER/OBSERVER in the log. Finally, you will see the thrift server
started log, and you can connect to FE through the MySQL client, which means FE
started successfully.
> 3. You can also check whether the startup is successful through the
following connection: http://fe_host:fe_http_port/api/bootstrap If it returns:
{"status":"OK","msg":"Success"}, it means the startup is successful, and the
rest , there may be a problem.
> 4. Visit http://fe_host:fe_http_port in the external environment to check
whether you can access the WebUI interface. The default login account is root
and the password is empty.
>
@@ -137,15 +145,15 @@ under the License.
Register BE to FE (using MySQL-Client, you need to install it yourself)
```shell
- # login
- mysql -h 127.0.0.1 -P 9030 -uroot
+ # login，Since it is a s
[GitHub] [doris] github-actions[bot] commented on pull request #11289: [feature-wip](unique-key-merge-on-write) update bitmap after compaction, DSIP-018
github-actions[bot] commented on PR #11289: URL: https://github.com/apache/doris/pull/11289#issuecomment-1197964878 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11289: [feature-wip](unique-key-merge-on-write) update bitmap after compaction, DSIP-018
github-actions[bot] commented on PR #11289: URL: https://github.com/apache/doris/pull/11289#issuecomment-1197964964 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #10792: [Feature][Materialized-View] support materialized view on vectorized engine
HappenLee commented on code in PR #10792: URL: https://github.com/apache/doris/pull/10792#discussion_r932062617 ## be/src/runtime/descriptors.cpp: ## @@ -339,6 +336,10 @@ RowDescriptor::RowDescriptor(const DescriptorTbl& desc_tbl, const std::vector
[doris-website] branch master updated: fix
This is an automated email from the ASF dual-hosted git repository. jiafengzheng pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris-website.git The following commit(s) were added to refs/heads/master by this push: new 202c05c59a8 fix 202c05c59a8 is described below commit 202c05c59a8c83624e1a09fc332e13458cd6b7f8 Author: jiafeng.zhang AuthorDate: Thu Jul 28 19:01:00 2022 +0800 fix --- sidebars.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sidebars.json b/sidebars.json index 53771fe5e9a..7e0a23fa5aa 100644 --- a/sidebars.json +++ b/sidebars.json @@ -915,7 +915,7 @@ "type": "category", "label": "Release notes", "items": [ -"releasenotes/release-1.1.0" +"releasenotes/release-1.1.0", "releasenotes/release-1.1.1" ] } - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener opened a new pull request, #11304: [community](actions): add `action` to check PR title
jackwener opened a new pull request, #11304: URL: https://github.com/apache/doris/pull/11304 # Proposed changes Issue Number: close #xxx ## Problem Summary: Add `action` to check PR title Title must be: [...](...) ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11294: [Doc] Add alter table comment doc
github-actions[bot] commented on PR #11294: URL: https://github.com/apache/doris/pull/11294#issuecomment-1198005652 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #11294: [Doc] Add alter table comment doc
github-actions[bot] commented on PR #11294: URL: https://github.com/apache/doris/pull/11294#issuecomment-1198005691 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] carlvinhust2012 commented on a diff in pull request #11213: [feature-wip](array-type) support the array type in reverse function
carlvinhust2012 commented on code in PR #11213:
URL: https://github.com/apache/doris/pull/11213#discussion_r932097679
##
docs/en/docs/sql-manual/sql-functions/string-functions/reverse.md:
##
@@ -52,5 +56,34 @@ mysql> SELECT REVERSE('你好');
+--+
1 row in set (0.00 sec)
```
+
+mysql> set enable_array_type=true;
+
+mysql> set enable_vectorized_engine=true;
+
+mysql> select k1, k2, reverse(k2) from array_test order by k1;
++--+-+-+
+| k1 | k2 | reverse(`k2`) |
++--+-+-+
+| 1 | [1, 2, 3, 4, 5] | [5, 4, 3, 2, 1] |
+| 2 | [6, 7, 8] | [8, 7, 6] |
+| 3 | [] | [] |
+| 4 | NULL| NULL|
+| 5 | [1, 2, 3, 4, 5, 4, 3, 2, 1] | [1, 2, 3, 4, 5, 4, 3, 2, 1] |
+| 6 | [1, 2, 3, NULL] | [NULL, 3, 2, 1] |
+| 7 | [4, 5, 6, NULL, NULL] | [NULL, NULL, 6, 5, 4] |
++--+-+-+
+
+mysql> select k1, k2, reverse(k2) from array_test01 order by k1;
++--+---+---+
+| k1 | k2| reverse(`k2`)
|
++--+---+---+
+| 1 | ['a', 'b', 'c', 'd'] | ['d', 'c', 'b', 'a']
|
+| 2 | ['e', 'f', 'g', 'h'] | ['h', 'g', 'f', 'e']
|
+| 3 | [NULL, 'a', NULL, 'b', NULL, 'c'] | ['c', NULL, 'b', NULL, 'a', NULL]
|
+| 4 | ['d', 'e', NULL, ' '] | [' ', NULL, 'e', 'd']
|
+| 5 | [' ', NULL, 'f', 'g'] | ['g', 'f', NULL, ' ']
|
++--+---+---+
+```
### keywords
REVERSE
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] caoliang-web commented on a diff in pull request #10905: [extension] Mysql database import doris by external tables
caoliang-web commented on code in PR #10905:
URL: https://github.com/apache/doris/pull/10905#discussion_r931830581
##
extension/mysql_to_doris/all_tables.sh:
##
@@ -0,0 +1,142 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+
+# This script is used to will mysql databases import doris by external
+
+
+#reference configuration file
+source ./conf/mysql.conf
+source ./conf/doris.conf
+
+#define mysql database and doris database
+d_mysql=$1
+d_doris=$2
+
+#check args
+if [ ! -n "$1" ];then
+echo "please check source database"
+exit
+fi
+if [ ! -n "$2" ];then
+echo "please check sink database"
+exit
+fi
+
+#mkdir files to store tables and tables.sql
+mkdir -p files
+rm -rf ./files/tables
+rm -rf ./files/tables.sql
+
+#get tables from mysql databases
+echo "use $d_mysql; show tables;" |mysql -h$mysql -uroot -p$mysql_password
2>/dev/null >> ./files/tables
+
+#delete tables first line
+sed -i '1d' ./files/tables
+
+#reference tables to create tables.sql
+for table in $(awk -F '\n' '{print $1}' ./files/tables)
+do
+sed -i "/${table}view/d" ./files/tables
+echo "use $d_mysql; show create table ${table};" |mysql -h$mysql
-uroot -p$mysql_password 2>/dev/null >> ./files/tables.sql
Review Comment:
The variable name in mysql.conf is mysql_host, and $mysql is used here,
which should be $mysql_host
##
extension/mysql_to_doris/all_tables.sh:
##
@@ -0,0 +1,142 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+
+# This script is used to will mysql databases import doris by external
+
+
+#reference configuration file
+source ./conf/mysql.conf
+source ./conf/doris.conf
+
+#define mysql database and doris database
+d_mysql=$1
+d_doris=$2
+
+#check args
+if [ ! -n "$1" ];then
+echo "please check source database"
+exit
+fi
+if [ ! -n "$2" ];then
+echo "please check sink database"
+exit
+fi
+
+#mkdir files to store tables and tables.sql
+mkdir -p files
+rm -rf ./files/tables
+rm -rf ./files/tables.sql
+
+#get tables from mysql databases
+echo "use $d_mysql; show tables;" |mysql -h$mysql -uroot -p$mysql_password
2>/dev/null >> ./files/tables
+
+#delete tables first line
+sed -i '1d' ./files/tables
+
+#reference tables to create tables.sql
+for table in $(awk -F '\n' '{print $1}' ./files/tables)
+do
+sed -i "/${table}view/d" ./files/tables
+echo "use $d_mysql; show create table ${table};" |mysql -h$mysql
-uroot -p$mysql_password 2>/dev/null >> ./files/tables.sql
+echo "print ${table} sql to tables.sql in the file dir"
+
+done
+
+echo '==start to transform mysql table for doris
extral table==='
+#adjust sql
+awk -F '\t' '{print $2}' ./files/tables.sql |awk '!(NR%2)' |awk '{print $0
";"}' > ./files/tables1.sql
+sed -i 's/\\n/\n/g' ./files/tables1.sql
+sed -n '/CREATE TABLE/,/ENGINE\=/p' ./files/tables1.sql > ./files/tables2.sql
+#delete tables special struct
+sed -i '/^ CON/d' ./files/tables2.sql
+sed -i '/^ KEY/d' ./files/tables2.sql
+rm -rf ./files/tables.sql
+rm -rf ./files/tables1.sql
+mv ./files/tables2.sql ./files/tables.sql
[GitHub] [doris] morrySnow commented on a diff in pull request #11304: [community](actions): add `action` to check PR title
morrySnow commented on code in PR #11304: URL: https://github.com/apache/doris/pull/11304#discussion_r932098944 ## .github/workflows/title-checker.yml: ## @@ -0,0 +1,33 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: 'Check PR title' + +on: + pull_request: +types: [opened, edited] + +jobs: + PR-title-checker: +name: "PR title checker" +runs-on: ubuntu-latest +steps: + - uses: ./.github/actions/action-pr-title +with: + regex: '\[([a-zA-Z])+\]\(([a-zA-Z])+\)(.*)' Review Comment: number should be support -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11304: [community](actions): add `action` to check PR title
morrySnow commented on code in PR #11304: URL: https://github.com/apache/doris/pull/11304#discussion_r932098944 ## .github/workflows/title-checker.yml: ## @@ -0,0 +1,33 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: 'Check PR title' + +on: + pull_request: +types: [opened, edited] + +jobs: + PR-title-checker: +name: "PR title checker" +runs-on: ubuntu-latest +steps: + - uses: ./.github/actions/action-pr-title +with: + regex: '\[([a-zA-Z])+\]\(([a-zA-Z])+\)(.*)' Review Comment: number and asterisk should be support -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11209: [feature] (Nereids) support limit clause
morrySnow commented on code in PR #11209:
URL: https://github.com/apache/doris/pull/11209#discussion_r932006408
##
fe/fe-core/src/test/java/org/apache/doris/nereids/parser/LimitClauseTest.java:
##
@@ -0,0 +1,93 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.parser;
+
+import org.apache.doris.nereids.trees.plans.logical.LogicalLimit;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+
+import org.junit.Assert;
+import org.junit.jupiter.api.Test;
+
+public class LimitClauseTest {
+@Test
+public void testLimit() {
+NereidsParser nereidsParser = new NereidsParser();
+String sql = "SELECT b FROM test order by a limit 3 offset 100";
+LogicalPlan logicalPlan = nereidsParser.parseSingle(sql);
+System.out.println(logicalPlan.treeString());
+Assert.assertTrue(logicalPlan instanceof LogicalLimit);
+LogicalLimit limit = (LogicalLimit) logicalPlan;
+Assert.assertEquals(3, limit.getLimit());
+Assert.assertEquals(100, limit.getOffset());
+Assert.assertEquals(1, limit.children().size());
+Assert.assertTrue(limit.child(0) instanceof LogicalSort);
+
+sql = "SELECT b FROM test order by a limit 100, 3";
+logicalPlan = nereidsParser.parseSingle(sql);
+System.out.println(logicalPlan.treeString());
+Assert.assertTrue(logicalPlan instanceof LogicalLimit);
+limit = (LogicalLimit) logicalPlan;
+Assert.assertEquals(3, limit.getLimit());
+Assert.assertEquals(100, limit.getOffset());
+Assert.assertEquals(1, limit.children().size());
+Assert.assertTrue(limit.child(0) instanceof LogicalSort);
+
+sql = "SELECT b FROM test limit 3";
+logicalPlan = nereidsParser.parseSingle(sql);
+System.out.println(logicalPlan.treeString());
+Assert.assertTrue(logicalPlan instanceof LogicalLimit);
+limit = (LogicalLimit) logicalPlan;
+Assert.assertEquals(3, limit.getLimit());
+Assert.assertEquals(0, limit.getOffset());
+Assert.assertEquals(1, limit.children().size());
+Assert.assertTrue(limit.child(0) instanceof LogicalProject);
+
+sql = "SELECT b FROM test order by a limit 3";
+logicalPlan = nereidsParser.parseSingle(sql);
+System.out.println(logicalPlan.treeString());
+Assert.assertTrue(logicalPlan instanceof LogicalLimit);
+limit = (LogicalLimit) logicalPlan;
+Assert.assertEquals(3, limit.getLimit());
+Assert.assertEquals(0, limit.getOffset());
+Assert.assertEquals(1, limit.children().size());
+Assert.assertTrue(limit.child(0) instanceof LogicalSort);
+}
+
+@Test
+public void testLimitExceptionCase() {
+NereidsParser nereidsParser = new NereidsParser();
+String sql = "SELECT b FROM test limit 3 offset 100";
+try {
+LogicalPlan logicalPlan = nereidsParser.parseSingle(sql);
+System.out.println(logicalPlan.treeString());
+} catch (IllegalStateException e) {
Review Comment:
if we want to check exception, please use assertThrow:
https://howtodoinjava.com/junit5/expected-exception-example/
##
fe/fe-core/src/test/java/org/apache/doris/nereids/parser/LimitClauseTest.java:
##
@@ -0,0 +1,93 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for th
[GitHub] [doris] github-actions[bot] commented on pull request #11189: [doc] update get-starting.md
github-actions[bot] commented on PR #11189: URL: https://github.com/apache/doris/pull/11189#issuecomment-1198032467 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on a diff in pull request #11222: [feature](nereids): polish property deriver enforcer job
jackwener commented on code in PR #11222:
URL: https://github.com/apache/doris/pull/11222#discussion_r932118198
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpecAny.java:
##
@@ -18,8 +18,16 @@
package org.apache.doris.nereids.properties;
/**
- * Describe random distribution.
+ * Data can be anywhere on the segments (required only).
*/
-public class RandomDistributionDesc extends DistributionSpec {
+public class DistributionSpecAny extends DistributionSpec {
+public DistributionSpecAny() {
Review Comment:
Use singleton is better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11035: [Feature](nereids) support sub-query and alias for TPC-H
morrySnow commented on code in PR #11035:
URL: https://github.com/apache/doris/pull/11035#discussion_r932117136
##
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##
@@ -206,11 +211,33 @@ public LogicalPlan
visitRegularQuerySpecification(RegularQuerySpecificationConte
/**
* Create an aliased table reference. This is typically used in FROM
clauses.
*/
+private LogicalPlan withTableAlias(LogicalPlan plan, TableAliasContext
ctx) {
Review Comment:
add a Developing annotation
##
fe/fe-core/src/main/java/org/apache/doris/nereids/memo/GroupExpression.java:
##
@@ -168,6 +169,12 @@ public boolean equals(Object o) {
return false;
}
GroupExpression that = (GroupExpression) o;
+// if the plan is LogicalRelation, this == that should be true,
+// when if one relation appear in plan more than once,
+// we cannot distinguish them throw equals function, since equals
function cannot use output info.
+if (plan instanceof LogicalRelation) {
Review Comment:
should also add physical relation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #11299: [enhancement](nereids) Don't normalize expressions when pushing down predicates
wangshuo128 commented on code in PR #11299:
URL: https://github.com/apache/doris/pull/11299#discussion_r932124737
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/NormalizeExpressions.java:
##
@@ -27,15 +28,16 @@
/**
* normalize expression of plan rule set.
*/
-public class NormalizeExpressionOfPlan extends ExpressionOfPlanRewrite {
+public class NormalizeExpressions extends ExpressionRewrite {
Review Comment:
```suggestion
public class NormalizeExpression extends ExpressionRewrite {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #11299: [enhancement](nereids) Don't normalize expressions when pushing down predicates
wangshuo128 commented on code in PR #11299:
URL: https://github.com/apache/doris/pull/11299#discussion_r932127219
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/OptimizeExpressions.java:
##
@@ -28,14 +27,13 @@
/**
* optimize expression of plan rule set.
*/
-public class OptimizeExpressionOfPlan extends ExpressionOfPlanRewrite {
+public class OptimizeExpressions extends ExpressionRewrite {
Review Comment:
```suggestion
public class OptimizeExpression extends ExpressionRewrite {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #11300: [feature](Nereids) Add subquery analyze
924060929 commented on code in PR #11300:
URL: https://github.com/apache/doris/pull/11300#discussion_r932049171
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalEnforceSingleRow.java:
##
@@ -0,0 +1,94 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * Guaranteed to return a result of 1 row.
Review Comment:
add comment:
```
e.g. select max(id) from tbl
```
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/Scope.java:
##
@@ -0,0 +1,49 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.analysis;
+
+import org.apache.doris.nereids.trees.expressions.Slot;
+
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * The slot range required for expression analyze.
+ */
+public class Scope {
+private final Optional outerScope;
+private final List slots;
+
+public Scope(Optional outerScope, List slots) {
+this.outerScope = outerScope;
+this.slots = slots;
+}
+
+public Scope(List slots) {
+this.outerScope = Optional.empty();
+this.slots = slots;
+}
+
+public List getSlots() {
+return slots;
+}
+
+public Optional getOuterScope() {
+return outerScope;
+}
Review Comment:
```suggestion
/** get scope link from inner to outer. */
public List toScopeLink() {
Scope scope = this;
Builder builder = ImmutableList.builder().add(scope);
while (scope.getOuterScope().isPresent()) {
scope = scope.getOuterScope().get();
builder.add(scope);
}
return builder.build();
}
```
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCorrelatedJoin.java:
##
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.proper
[GitHub] [doris] wangshuo128 commented on a diff in pull request #11299: [enhancement](nereids) Don't normalize expressions when pushing down predicates
wangshuo128 commented on code in PR #11299:
URL: https://github.com/apache/doris/pull/11299#discussion_r932133075
##
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/PushDownPredicateTest.java:
##
@@ -242,6 +243,7 @@ public void pushDownPredicateIntoScanTest4() {
}
private Memo rewrite(Plan plan) {
-return PlanRewriter.topDownRewriteMemo(plan, new ConnectContext(), new
PushPredicateThroughJoin());
+Plan normalizedPlan = PlanRewriter.topDownRewrite(plan, new
ConnectContext(), new NormalizeExpressions());
Review Comment:
You could add a new interface in `PlanRewriter` like
`topDownRewriteMemo(Plan plan, ConnectContext connectContext,
List> ruleBatches)` to run apply rules in batches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #11299: [enhancement](nereids) Don't normalize expressions when pushing down predicates
wangshuo128 commented on code in PR #11299:
URL: https://github.com/apache/doris/pull/11299#discussion_r932133075
##
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/logical/PushDownPredicateTest.java:
##
@@ -242,6 +243,7 @@ public void pushDownPredicateIntoScanTest4() {
}
private Memo rewrite(Plan plan) {
-return PlanRewriter.topDownRewriteMemo(plan, new ConnectContext(), new
PushPredicateThroughJoin());
+Plan normalizedPlan = PlanRewriter.topDownRewrite(plan, new
ConnectContext(), new NormalizeExpressions());
Review Comment:
You could add a new interface in `PlanRewriter` like
`topDownRewriteMemo(Plan plan, ConnectContext connectContext,
List> ruleBatches)` to apply rules in batches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangshuo128 commented on pull request #11299: [enhancement](nereids) Don't normalize expressions when pushing down predicates
wangshuo128 commented on PR #11299: URL: https://github.com/apache/doris/pull/11299#issuecomment-1198052836 I suggest renaming the PR title to "Normalize expressions before performing plan rewriting". The current title is a bit misleading. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Wilson-BT opened a new issue, #11305: [Bug] Only suffort '\n' for line delimiter when streamload
Wilson-BT opened a new issue, #11305: URL: https://github.com/apache/doris/issues/11305 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Version 1.1.0 ### What's Wrong? For now, when we use http inputstream to do stream load, we can only use \n as line delimiter，otherwise, doris will return json line parse error.if one column has '\n' internal，it will cause another line parse error. i think we should think about it to avoid them. ### What You Expected? we can use another special characters for line delimiter. ### How to Reproduce? _No response_ ### Anything Else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow commented on a diff in pull request #11262: [feature] (Nereids) add rule to merge consecutive project nodes
morrySnow commented on code in PR #11262:
URL: https://github.com/apache/doris/pull/11262#discussion_r932134296
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/MergeConsecutiveProjects.java:
##
@@ -0,0 +1,99 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import
org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+import org.apache.doris.nereids.trees.expressions.visitor.IterationVisitor;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * this rule aims to merge consecutive filters.
+ * For example:
+ * logical plan tree:
+ *project(a)
+ * |
+ *project(a,b)
+ * |
+ *project(a, b, c)
+ * |
+ *scan
+ * transformed to:
+ *project(a)
+ * |
+ * scan
+ */
+
+public class MergeConsecutiveProjects extends OneRewriteRuleFactory {
+@Override
+public Rule build() {
+return logicalProject(logicalProject()).then(project -> {
+List projectExpressions = project.getProjects();
+LogicalProject childProject = project.child();
+List childProjectExpressions =
childProject.getProjects();
+HashMap childAliasMap = new HashMap();
+AliasExtractor extractor = new AliasExtractor();
+for (NamedExpression expression : childProjectExpressions) {
+extractor.visit(expression, childAliasMap);
+}
+ExpressionReplacer replacer =
+new ExpressionReplacer>();
+projectExpressions = projectExpressions.stream()
+.map(e -> replacer.visit(e, childAliasMap))
+.map(NamedExpression.class::cast)
+.collect(Collectors.toList());
+return new LogicalProject(projectExpressions, (Plan)
childProject.children().get(0));
+}).toRule(RuleType.MERGE_CONSECUTIVE_PROJECTS);
+}
+
+private class ExpressionReplacer
Review Comment:
move out ExpressionReplacer from AggregateDisassemble to
tree/expressions/visitor and use it here
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/MergeConsecutiveProjects.java:
##
@@ -0,0 +1,99 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+imp
[GitHub] [doris] carlvinhust2012 opened a new pull request, #11306: [refactor](array-type) optimize the array_distinct and array_sort fun…
carlvinhust2012 opened a new pull request, #11306: URL: https://github.com/apache/doris/pull/11306 # Proposed changes 1. this pr is used to optimize the array_distinct and array_sort function. Issue Number: close #10052 ## Problem Summary: ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 3. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 4. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 5. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 6. Does it need to update dependencies: - [ ] Yes - [ ] No 7. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Wilson-BT commented on issue #11258: [Feature] JSON data is dynamically written to the Doris table
Wilson-BT commented on issue #11258: URL: https://github.com/apache/doris/issues/11258#issuecomment-1198066436 > > Yes, encapsulate such a data structure, dynamically load by spelling the URL on the sink side, and add a keyby operator before sink > > At present, flink-doris-connector initiates the stream load of the table when the flink task starts, instead of doing the stream load when the upstream data is received. How to do this dynamic stream load? Please describe your design in detail～ Many users put all the canal logs of all tables in the business library into one topic, which needs to be distributed before they can use doris-flink-connector. His idea is to edit a task to synchronize the entire library. Because currently doris-flink-connector uses http inputstream, that is, a checkpoint opens a stream, and a streamLoad url is strongly bound. In this case, we can only use the flink side to cache data, and then a table generates a buffer, and bind the corresponding table-streamload-url, set a threshold, such as rows number or batch size to submit tasks, just like doris-datax-writer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Wilson-BT commented on issue #11258: [Feature] JSON data is dynamically written to the Doris table
Wilson-BT commented on issue #11258: URL: https://github.com/apache/doris/issues/11258#issuecomment-1198067162 Many users put all the canal logs of all tables in the business library into one topic, which needs to be distributed before they can use doris-flink-connector. His idea is to edit a task to synchronize the entire library. Because currently doris-flink-connector uses http inputstream, that is, a checkpoint opens a stream, and a streamLoad url is strongly bound. In this case, we can only use the flink side to cache data, and then a table generates a buffer, and bind the corresponding table-streamload-url, set a threshold, such as rows number or batch size to submit tasks, just like doris-datax-writer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #11248: [feature] (Nereids) add rewrite rule to merge consecutive filter nodes
924060929 commented on code in PR #11248:
URL: https://github.com/apache/doris/pull/11248#discussion_r932145742
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/MergeConsecutiveFilters.java:
##
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import java.util.Optional;
+
+/**
+ * this rule aims to merge consecutive filters.
+ * For example:
+ * logical plan tree:
+ * project
+ * |
+ *filter(a>0)
+ * |
+ *filter(b>0)
+ * |
+ *scan
+ * transformed to:
+ *project
+ * |
+ *filter(a>0 and b>0)
+ * |
+ * scan
+ */
+public class MergeConsecutiveFilters extends OneRewriteRuleFactory {
+@Override
+public Rule build() {
+return logicalFilter(logicalFilter()).then(filter -> {
+LogicalFilter childFilter = filter.child();
+Expression predicates = filter.getPredicates();
+Expression childPredicates = childFilter.getPredicates();
+Expression mergedPredicates = ExpressionUtils.and(predicates,
childPredicates);
+LogicalFilter mergedFilter = new LogicalFilter(
+mergedPredicates,
+Optional.empty(),
+Optional.of(filter.getLogicalProperties()),
Review Comment:
this line will skip copyIn, so this transform do nothing.
I think you need
```java
return new LogicalFilter(mergedPredicates, childFilter.child());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Wilson-BT commented on issue #11258: [Feature] JSON data is dynamically written to the Doris table
Wilson-BT commented on issue #11258: URL: https://github.com/apache/doris/issues/11258#issuecomment-1198067658 > > Yes, encapsulate such a data structure, dynamically load by spelling the URL on the sink side, and add a keyby operator before sink > > At present, flink-doris-connector initiates the stream load of the table when the flink task starts, instead of doing the stream load when the upstream data is received. How to do this dynamic stream load? Please describe your design in detail～ Many users put all the canal logs of all tables in the business library into one topic, which needs to be distributed before they can use doris-flink-connector. His idea is to edit a task to synchronize the entire library. Because currently doris-flink-connector uses http inputstream, that is, a checkpoint opens a stream, and a streamLoad url is strongly bound. In this case, we can only use the flink side to cache data, and then a table generates a buffer, and bind the corresponding table-streamload-url, set a threshold, such as rows number or batch size to submit tasks, just like doris-datax-writer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #11248: [feature] (Nereids) add rewrite rule to merge consecutive filter nodes
924060929 commented on code in PR #11248:
URL: https://github.com/apache/doris/pull/11248#discussion_r932145742
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/MergeConsecutiveFilters.java:
##
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import java.util.Optional;
+
+/**
+ * this rule aims to merge consecutive filters.
+ * For example:
+ * logical plan tree:
+ * project
+ * |
+ *filter(a>0)
+ * |
+ *filter(b>0)
+ * |
+ *scan
+ * transformed to:
+ *project
+ * |
+ *filter(a>0 and b>0)
+ * |
+ * scan
+ */
+public class MergeConsecutiveFilters extends OneRewriteRuleFactory {
+@Override
+public Rule build() {
+return logicalFilter(logicalFilter()).then(filter -> {
+LogicalFilter childFilter = filter.child();
+Expression predicates = filter.getPredicates();
+Expression childPredicates = childFilter.getPredicates();
+Expression mergedPredicates = ExpressionUtils.and(predicates,
childPredicates);
+LogicalFilter mergedFilter = new LogicalFilter(
+mergedPredicates,
+Optional.empty(),
+Optional.of(filter.getLogicalProperties()),
Review Comment:
re-use the logical properties will skip copyIn, so this transform do nothing.
I think you need
```java
return new LogicalFilter(mergedPredicates, childFilter.child());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on a diff in pull request #11222: [feature](nereids): polish property deriver enforcer job
jackwener commented on code in PR #11222:
URL: https://github.com/apache/doris/pull/11222#discussion_r932154104
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpec.java:
##
@@ -21,52 +21,38 @@
import org.apache.doris.nereids.memo.GroupExpression;
import org.apache.doris.nereids.trees.plans.GroupPlan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalDistribution;
-import org.apache.doris.planner.DataPartition;
import com.google.common.collect.Lists;
/**
* Spec of data distribution.
+ * GPORCA has more type in CDistributionSpec.
*/
-public class DistributionSpec {
-
-private DataPartition dataPartition;
-
-// TODO: why exist?
-public DistributionSpec() {
-}
-
-public DistributionSpec(DataPartition dataPartition) {
-this.dataPartition = dataPartition;
-}
-
+public abstract class DistributionSpec {
/**
* TODO: need read ORCA.
* Whether other `DistributionSpec` is satisfied the current
`DistributionSpec`.
*
* @param other another DistributionSpec.
*/
-public boolean meet(DistributionSpec other) {
-return false;
-}
-
-public DataPartition getDataPartition() {
-return dataPartition;
-}
-
-public void setDataPartition(DataPartition dataPartition) {
-this.dataPartition = dataPartition;
-}
+public abstract boolean meet(DistributionSpec other);
public GroupExpression addEnforcer(Group child) {
PhysicalDistribution distribution = new PhysicalDistribution(
-new DistributionSpec(dataPartition),
child.getLogicalProperties(), new GroupPlan(child));
+this,
+child.getLogicalProperties(),
Review Comment:
good catch, but I haven't figured out how to do it yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #11307: [doc][refactor](metrics) Reorganize FE metrics and add document
morningman opened a new pull request, #11307: URL: https://github.com/apache/doris/pull/11307 # Proposed changes Issue Number: close #xxx ## Problem Summary: ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [ ] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [ ] No Need 5. Does it need to update dependencies: - [ ] Yes - [ ] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [ ] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #11248: [feature] (Nereids) add rewrite rule to merge consecutive filter nodes
924060929 commented on code in PR #11248:
URL: https://github.com/apache/doris/pull/11248#discussion_r932145742
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/MergeConsecutiveFilters.java:
##
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+
+import java.util.Optional;
+
+/**
+ * this rule aims to merge consecutive filters.
+ * For example:
+ * logical plan tree:
+ * project
+ * |
+ *filter(a>0)
+ * |
+ *filter(b>0)
+ * |
+ *scan
+ * transformed to:
+ *project
+ * |
+ *filter(a>0 and b>0)
+ * |
+ * scan
+ */
+public class MergeConsecutiveFilters extends OneRewriteRuleFactory {
+@Override
+public Rule build() {
+return logicalFilter(logicalFilter()).then(filter -> {
+LogicalFilter childFilter = filter.child();
+Expression predicates = filter.getPredicates();
+Expression childPredicates = childFilter.getPredicates();
+Expression mergedPredicates = ExpressionUtils.and(predicates,
childPredicates);
+LogicalFilter mergedFilter = new LogicalFilter(
+mergedPredicates,
+Optional.empty(),
+Optional.of(filter.getLogicalProperties()),
Review Comment:
re-use the logical properties is useless, I think you need
```java
return new LogicalFilter(mergedPredicates, childFilter.child());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt opened a new pull request, #11308: [Bug][Function] core dump on sum(distinct)
BiteThet opened a new pull request, #11308: URL: https://github.com/apache/doris/pull/11308 # Proposed changes Issue Number: close #11302 ## Problem Summary: ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [X] Fix - [ ] Feature-WIP - [ ] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [X] No - [ ] I don't know 3. Has unit tests been added: - [X] Yes - [ ] No - [ ] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [X] No Need 5. Does it need to update dependencies: - [ ] Yes - [X] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [X] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Toms1999 commented on a diff in pull request #10905: [extension] Mysql database import doris by external tables
Toms1999 commented on code in PR #10905:
URL: https://github.com/apache/doris/pull/10905#discussion_r932158162
##
extension/mysql_to_doris/all_tables.sh:
##
@@ -0,0 +1,142 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+
+# This script is used to will mysql databases import doris by external
+
+
+#reference configuration file
+source ./conf/mysql.conf
+source ./conf/doris.conf
+
+#define mysql database and doris database
+d_mysql=$1
+d_doris=$2
+
+#check args
+if [ ! -n "$1" ];then
+echo "please check source database"
+exit
+fi
+if [ ! -n "$2" ];then
+echo "please check sink database"
+exit
+fi
+
+#mkdir files to store tables and tables.sql
+mkdir -p files
+rm -rf ./files/tables
+rm -rf ./files/tables.sql
+
+#get tables from mysql databases
+echo "use $d_mysql; show tables;" |mysql -h$mysql -uroot -p$mysql_password
2>/dev/null >> ./files/tables
+
+#delete tables first line
+sed -i '1d' ./files/tables
+
+#reference tables to create tables.sql
+for table in $(awk -F '\n' '{print $1}' ./files/tables)
+do
+sed -i "/${table}view/d" ./files/tables
+echo "use $d_mysql; show create table ${table};" |mysql -h$mysql
-uroot -p$mysql_password 2>/dev/null >> ./files/tables.sql
Review Comment:
you are right，I've already fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Wilson-BT commented on issue #11258: [Feature] JSON data is dynamically written to the Doris table
Wilson-BT commented on issue #11258:
URL: https://github.com/apache/doris/issues/11258#issuecomment-1198078387
> 生成
canal json ---> serialize to RowData -->
TableBufferMap> map
key is ```{db}_{table}```, buffer value is a buffer contained```
{"column_a":"value_a","column_b":"value_b"...}```
we can submit to http://xxx:xx/api/{db}/{table}/_streamLoad when buffer over
size like doris-datax-writer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Toms1999 commented on a diff in pull request #10905: [extension] Mysql database import doris by external tables
Toms1999 commented on code in PR #10905:
URL: https://github.com/apache/doris/pull/10905#discussion_r932160212
##
extension/mysql_to_doris/all_tables.sh:
##
@@ -0,0 +1,142 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+
+# This script is used to will mysql databases import doris by external
+
+
+#reference configuration file
+source ./conf/mysql.conf
+source ./conf/doris.conf
+
+#define mysql database and doris database
+d_mysql=$1
+d_doris=$2
+
+#check args
+if [ ! -n "$1" ];then
+echo "please check source database"
+exit
+fi
+if [ ! -n "$2" ];then
+echo "please check sink database"
+exit
+fi
+
+#mkdir files to store tables and tables.sql
+mkdir -p files
+rm -rf ./files/tables
+rm -rf ./files/tables.sql
+
+#get tables from mysql databases
+echo "use $d_mysql; show tables;" |mysql -h$mysql -uroot -p$mysql_password
2>/dev/null >> ./files/tables
+
+#delete tables first line
+sed -i '1d' ./files/tables
+
+#reference tables to create tables.sql
+for table in $(awk -F '\n' '{print $1}' ./files/tables)
+do
+sed -i "/${table}view/d" ./files/tables
+echo "use $d_mysql; show create table ${table};" |mysql -h$mysql
-uroot -p$mysql_password 2>/dev/null >> ./files/tables.sql
+echo "print ${table} sql to tables.sql in the file dir"
+
+done
+
+echo '==start to transform mysql table for doris
extral table==='
+#adjust sql
+awk -F '\t' '{print $2}' ./files/tables.sql |awk '!(NR%2)' |awk '{print $0
";"}' > ./files/tables1.sql
+sed -i 's/\\n/\n/g' ./files/tables1.sql
+sed -n '/CREATE TABLE/,/ENGINE\=/p' ./files/tables1.sql > ./files/tables2.sql
+#delete tables special struct
+sed -i '/^ CON/d' ./files/tables2.sql
+sed -i '/^ KEY/d' ./files/tables2.sql
+rm -rf ./files/tables.sql
+rm -rf ./files/tables1.sql
+mv ./files/tables2.sql ./files/tables.sql
+
+#start transform tables struct
+sed -i '/ENGINE=/a) ENGINE=MYSQL\n COMMENT "MYSQL"\nPROPERTIES (\n"host" =
"ApacheDorisHostIp",\n"port" = "3306",\n"user" = "root",\n"password" =
"ApacheDorisHostPassword",\n"database" = "ApacheDorisDataBases",\n"table" =
"ApacheDorisTables");' ./files/tables.sql
+
+#delete match line
+sed -i '/ENGINT=/d' ./files/tables.sql
+sed -i '/PRIMARY KEY/d' ./files/tables.sql
+sed -i '/UNIQUE KEY/d' ./files/tables.sql
+#delete , at the beginning (
+sed -i '/,\s*$/{:loop; N; /,\(\s*\|\n\))/! bloop; s/,\s*[\n]\?\s*)/\n)/}'
./files/tables.sql
+
+#delete a line on keyword
+sed -i -e '$!N;/\n.*ENGINE=MYSQL/!P;D' ./files/tables.sql
+#replace mysql password、database、table、host
+for t_name in $(awk -F '\n' '{print $1}' ./files/tables)
+do
+sed -i "0,/ApacheDorisHostIp/s/ApacheDorisHostIp/${mysql}/"
./files/tables.sql
+sed -i
"0,/ApacheDorisHostPassword/s/ApacheDorisHostPassword/${mysql_password}/"
./files/tables.sql
+sed -i "0,/ApacheDorisDataBases/s/ApacheDorisDataBases/${d_mysql}/"
./files/tables.sql
+sed -i "0,/ApacheDorisTables/s/ApacheDorisTables/${t_name}/"
./files/tables.sql
+
+done
+##
+#replace mysql type with doris
+sed -i 's/text/string/g' ./files/tables.sql
Review Comment:
i will string replace with varchar(65535),but some data type not a good
idea,I hope to improve it in the future by issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zhannngchen commented on a diff in pull request #11289: [feature-wip](unique-key-merge-on-write) update bitmap after compaction, DSIP-018
zhannngchen commented on code in PR #11289:
URL: https://github.com/apache/doris/pull/11289#discussion_r932107099
##
be/src/olap/tablet_meta.cpp:
##
@@ -734,6 +734,45 @@ Status TabletMeta::set_partition_id(int64_t partition_id) {
return Status::OK();
}
+// update dst rowset delete bitmap for unique key merge on write table.
+// we take a delete bitmap's snapshot of origin rowset, but the
Review Comment:
we take a delete bitmap's snapshot of origin rowset at the beginning of
compaction
##
be/src/olap/tablet_meta.cpp:
##
@@ -734,6 +734,45 @@ Status TabletMeta::set_partition_id(int64_t partition_id) {
return Status::OK();
}
+// update dst rowset delete bitmap for unique key merge on write table.
+// we take a delete bitmap's snapshot of origin rowset, but the
+// detele bitmap of origin rowset may update when compaction.
+// some key deleted on compaction will exist in dst rowset.
Review Comment:
some key might be deleted during compaction, which would exist in dst rowset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on a diff in pull request #11222: [feature](nereids): polish property deriver enforcer job
jackwener commented on code in PR #11222:
URL: https://github.com/apache/doris/pull/11222#discussion_r932165334
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpec.java:
##
@@ -21,52 +21,38 @@
import org.apache.doris.nereids.memo.GroupExpression;
import org.apache.doris.nereids.trees.plans.GroupPlan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalDistribution;
-import org.apache.doris.planner.DataPartition;
import com.google.common.collect.Lists;
/**
* Spec of data distribution.
+ * GPORCA has more type in CDistributionSpec.
*/
-public class DistributionSpec {
-
-private DataPartition dataPartition;
-
-// TODO: why exist?
-public DistributionSpec() {
-}
-
-public DistributionSpec(DataPartition dataPartition) {
-this.dataPartition = dataPartition;
-}
-
+public abstract class DistributionSpec {
/**
* TODO: need read ORCA.
* Whether other `DistributionSpec` is satisfied the current
`DistributionSpec`.
*
* @param other another DistributionSpec.
*/
-public boolean meet(DistributionSpec other) {
-return false;
-}
-
-public DataPartition getDataPartition() {
-return dataPartition;
-}
-
-public void setDataPartition(DataPartition dataPartition) {
-this.dataPartition = dataPartition;
-}
+public abstract boolean meet(DistributionSpec other);
Review Comment:
I add comment, and have renamed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on a diff in pull request #11222: [feature](nereids): polish property deriver enforcer job
jackwener commented on code in PR #11222: URL: https://github.com/apache/doris/pull/11222#discussion_r932165968 ## fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpecGather.java: ## @@ -20,9 +20,14 @@ /** * Re-shuffle. Review Comment: Yes, it's history comment, and it's outdated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on a diff in pull request #11222: [feature](nereids): polish property deriver enforcer job
jackwener commented on code in PR #11222:
URL: https://github.com/apache/doris/pull/11222#discussion_r932167685
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpecAny.java:
##
@@ -18,8 +18,16 @@
package org.apache.doris.nereids.properties;
/**
- * Describe random distribution.
+ * Data can be anywhere on the segments (required only).
*/
-public class RandomDistributionDesc extends DistributionSpec {
+public class DistributionSpecAny extends DistributionSpec {
+public DistributionSpecAny() {
+super();
+}
+
+@Override
+public boolean meet(DistributionSpec other) {
+return other instanceof DistributionSpecAny;
Review Comment:
super's function's comment is wrong, I have fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow opened a new pull request, #11309: [feature](planner)add projection info TPlanNode to support projection on ExecNode
morrySnow opened a new pull request, #11309: URL: https://github.com/apache/doris/pull/11309 # Proposed changes Issue Number: close #xxx ## Problem Summary: ## Checklist(Required) 1. Type of your changes: - [ ] Improvement - [ ] Fix - [ ] Feature-WIP - [x] Feature - [ ] Doc - [ ] Refator - [ ] Others: 2. Does it affect the original behavior: - [ ] Yes - [x] No - [ ] I don't know 3. Has unit tests been added: - [ ] Yes - [ ] No - [x] No Need 4. Has document been added or modified: - [ ] Yes - [ ] No - [x] No Need 5. Does it need to update dependencies: - [ ] Yes - [x] No 6. Are there any changes that cannot be rolled back: - [ ] Yes - [x] No ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on a diff in pull request #11222: [feature](nereids): polish property deriver enforcer job
jackwener commented on code in PR #11222:
URL: https://github.com/apache/doris/pull/11222#discussion_r932173805
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpec.java:
##
@@ -21,52 +21,38 @@
import org.apache.doris.nereids.memo.GroupExpression;
import org.apache.doris.nereids.trees.plans.GroupPlan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalDistribution;
-import org.apache.doris.planner.DataPartition;
import com.google.common.collect.Lists;
/**
* Spec of data distribution.
+ * GPORCA has more type in CDistributionSpec.
*/
-public class DistributionSpec {
-
-private DataPartition dataPartition;
-
-// TODO: why exist?
-public DistributionSpec() {
-}
-
-public DistributionSpec(DataPartition dataPartition) {
-this.dataPartition = dataPartition;
-}
-
+public abstract class DistributionSpec {
/**
* TODO: need read ORCA.
* Whether other `DistributionSpec` is satisfied the current
`DistributionSpec`.
*
* @param other another DistributionSpec.
*/
-public boolean meet(DistributionSpec other) {
-return false;
-}
-
-public DataPartition getDataPartition() {
-return dataPartition;
-}
-
-public void setDataPartition(DataPartition dataPartition) {
-this.dataPartition = dataPartition;
-}
+public abstract boolean meet(DistributionSpec other);
public GroupExpression addEnforcer(Group child) {
PhysicalDistribution distribution = new PhysicalDistribution(
-new DistributionSpec(dataPartition),
child.getLogicalProperties(), new GroupPlan(child));
+this,
+child.getLogicalProperties(),
Review Comment:
Add a todo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Im-Manshushu commented on issue #11258: [Feature] JSON data is dynamically written to the Doris table
Im-Manshushu commented on issue #11258: URL: https://github.com/apache/doris/issues/11258#issuecomment-1198094741 What you said is quite right. My idea is like this. The previous answer may not be described clearly -- 原始邮件 -- 发件人: "apache/doris" ***@***.***>; 发送时间: 2022年7月28日(星期四) 晚上8:20 ***@***.***>; 抄送: "I'm ~ ***@***.**@***.***>; 主题: Re: [apache/doris] [Feature] JSON data is dynamically written to the Doris table (Issue #11258) Yes, encapsulate such a data structure, dynamically load by spelling the URL on the sink side, and add a keyby operator before sink At present, flink-doris-connector initiates the stream load of the table when the flink task starts, instead of doing the stream load when the upstream data is received. How to do this dynamic stream load? Please describe your design in detail～ Many users put all the canal logs of all tables in the business library into one topic, which needs to be distributed before they can use doris-flink-connector. His idea is to edit a task to synchronize the entire library. Because currently doris-flink-connector uses http inputstream, that is, a checkpoint opens a stream, and a streamLoad url is strongly bound. In this case, we can only use the flink side to cache data, and then a table generates a buffer, and bind the corresponding table-streamload-url, set a threshold, such as rows number or batch size to submit tasks, just like doris-datax-writer. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #11209: [feature] (Nereids) support limit clause
924060929 commented on code in PR #11209:
URL: https://github.com/apache/doris/pull/11209#discussion_r932170082
##
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##
@@ -532,13 +534,23 @@ public List
visitNamedExpressionSeq(NamedExpressionSeqContext namedC
* @return List of OrderKey
*/
@Override
-public List visitQueryOrganization(QueryOrganizationContext ctx)
{
+public QueryOrganization visitQueryOrganization(QueryOrganizationContext
ctx) {
return ParserUtils.withOrigin(ctx, () -> {
+List orderKeys;
if (ctx.sortClause().ORDER() != null) {
-return visit(ctx.sortClause().sortItem(), OrderKey.class);
+orderKeys = visit(ctx.sortClause().sortItem(), OrderKey.class);
} else {
-return ImmutableList.of();
+orderKeys = ImmutableList.of();
}
+long limit = Long.MAX_VALUE;
Review Comment:
the limitClause definition above can ensure this limit must has a value.
If you keep the definition as before, I think this limit should be
`Optional.empty()` and return originPlan without wrap a QueryOrganization.
##
fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4:
##
@@ -119,6 +119,12 @@ sortItem
: expression ordering = (ASC | DESC)?
;
+limitClause
+: (LIMIT limit=INTEGER_VALUE)?
+| (LIMIT limit=INTEGER_VALUE OFFSET offset=INTEGER_VALUE)?
+| (LIMIT offset=INTEGER_VALUE COMMA limit=INTEGER_VALUE)?
+;
Review Comment:
why not set sortClause and limitClause to optional?
```antlr4
queryOrganization
: sortClause? limitClause?
;
sortItem
: expression ordering = (ASC | DESC)
;
limitClause
: (LIMIT limit=INTEGER_VALUE)
| (LIMIT limit=INTEGER_VALUE OFFSET offset=INTEGER_VALUE)
| (LIMIT offset=INTEGER_VALUE COMMA limit=INTEGER_VALUE)
;
```
##
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##
@@ -532,13 +534,23 @@ public List
visitNamedExpressionSeq(NamedExpressionSeqContext namedC
* @return List of OrderKey
*/
@Override
-public List visitQueryOrganization(QueryOrganizationContext ctx)
{
+public QueryOrganization visitQueryOrganization(QueryOrganizationContext
ctx) {
return ParserUtils.withOrigin(ctx, () -> {
+List orderKeys;
if (ctx.sortClause().ORDER() != null) {
-return visit(ctx.sortClause().sortItem(), OrderKey.class);
+orderKeys = visit(ctx.sortClause().sortItem(), OrderKey.class);
} else {
-return ImmutableList.of();
+orderKeys = ImmutableList.of();
}
+long limit = Long.MAX_VALUE;
+long offset = 0;
+if (ctx.limitClause().LIMIT() != null) {
+limit = Long.parseLong(ctx.limitClause().limit.getText());
+if (ctx.limitClause().OFFSET() != null ||
ctx.limitClause().COMMA() != null) {
+offset =
Long.parseLong(ctx.limitClause().offset.getText());
+}
+}
+return new QueryOrganization(orderKeys, limit, offset);
Review Comment:
I think QueryOrganization is should be divided into LogicalSort and
LogicalLimit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jackwener commented on a diff in pull request #11222: [feature](nereids): polish property deriver enforcer job
jackwener commented on code in PR #11222:
URL: https://github.com/apache/doris/pull/11222#discussion_r932175672
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/RequestPropertyDeriver.java:
##
@@ -0,0 +1,127 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.properties;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.PlanContext;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.DistributionSpecHash.ShuffleType;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.physical.PhysicalHashJoin;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+
+/**
+ * Used for parent property drive.
+ */
+public class RequestPropertyDeriver extends PlanVisitor {
+/*
+ * requestPropertyFromParent
+ * │
+ * ▼
+ * curNode (current plan node in current CostAndEnforcerJob)
+ * │
+ * ▼
+ * requestPropertyToChildren
+ */
+PhysicalProperties requestPropertyFromParent;
+List> requestPropertyToChildren;
+
+public RequestPropertyDeriver(JobContext context) {
+this.requestPropertyFromParent = context.getRequiredProperties();
+}
+
+public List>
getRequiredPropertyListList(GroupExpression groupExpression) {
+requestPropertyToChildren = Lists.newArrayList();
+groupExpression.getPlan().accept(this, new
PlanContext(groupExpression));
+return requestPropertyToChildren;
+}
+
+@Override
+public Void visit(Plan plan, PlanContext context) {
+List requiredPropertyList = Lists.newArrayList();
+for (int i = 0; i < context.getGroupExpression().arity(); i++) {
+requiredPropertyList.add(new PhysicalProperties());
+}
+requestPropertyToChildren.add(requiredPropertyList);
+return null;
+}
+
+@Override
+public Void visitPhysicalHashJoin(PhysicalHashJoin hashJoin,
PlanContext context) {
+// for broadcast join
+List propertiesForBroadcast = Lists.newArrayList(
+new PhysicalProperties(),
+new PhysicalProperties(new DistributionSpecReplicated())
+);
+// for shuffle join
+Pair, List> onClauseUsedSlots =
JoinUtils.getOnClauseUsedSlots(hashJoin);
+List propertiesForShuffle = Lists.newArrayList(
+new PhysicalProperties(new
DistributionSpecHash(onClauseUsedSlots.first, ShuffleType.JOIN)),
+new PhysicalProperties(new
DistributionSpecHash(onClauseUsedSlots.second, ShuffleType.JOIN)));
+
+if (!JoinUtils.onlyBroadcast(hashJoin)) {
+requestPropertyToChildren.add(propertiesForShuffle);
+}
+if (!JoinUtils.onlyShuffle(hashJoin)) {
+requestPropertyToChildren.add(propertiesForBroadcast);
+}
+
+return null;
+}
+
+protected static List
computeShuffleJoinRequiredProperties(
Review Comment:
Maybe we we will it in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Im-Manshushu commented on issue #11258: [Feature] JSON data is dynamically written to the Doris table
Im-Manshushu commented on issue #11258:
URL: https://github.com/apache/doris/issues/11258#issuecomment-1198096957
Yes, that's it. A flink task can synchronize the data of the entire database
-- 原始邮件 --
发件人:
"apache/doris"
***@***.***>;
发送时间: 2022年7月28日(星期四) 晚上8:31
***@***.***>;
抄送: "I'm ~ ***@***.**@***.***>;
主题: Re: [apache/doris] [Feature] JSON data is dynamically written to
the Doris table (Issue #11258)
生成
canal json ---> serialize to RowData -->
TableBufferMaphttp://xxx:xx/api/{db}/{table}/_streamLoad when buffer
over size like doris-datax-writer.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
