Re: [PR] [Compatibility](agg-state) add be_exec_version check to avoid aggregation data format changed le… [doris]
hello-stephen commented on PR #40193: URL: https://github.com/apache/doris/pull/40193#issuecomment-2325747172 @HappenLee PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [wip][enhancement](cloud) clarify codes and make TTL expiration work after abnormal cache type transition [doris]
github-actions[bot] commented on code in PR #40226:
URL: https://github.com/apache/doris/pull/40226#discussion_r1741530477
##
be/src/io/cache/fs_file_cache_storage.cpp:
##
@@ -227,7 +286,7 @@ std::string
FSFileCacheStorage::get_path_in_local_cache(const UInt128Wrapper& va
}
}
-Status FSFileCacheStorage::rebuild_data_structure() const {
+Status FSFileCacheStorage::upgrade_cache_dir_if_necessary() const {
Review Comment:
warning: function 'upgrade_cache_dir_if_necessary' has cognitive complexity
of 119 (threshold 50) [readability-function-cognitive-complexity]
```cpp
Status FSFileCacheStorage::upgrade_cache_dir_if_necessary() const {
^
```
Additional context
**be/src/io/cache/fs_file_cache_storage.cpp:292:** +1, including nesting
penalty of 0, nesting level increased to 1
```cpp
RETURN_IF_ERROR(read_file_cache_version(&version));
^
```
**be/src/common/status.h:628:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do {\
^
```
**be/src/io/cache/fs_file_cache_storage.cpp:292:** +2, including nesting
penalty of 1, nesting level increased to 2
```cpp
RETURN_IF_ERROR(read_file_cache_version(&version));
^
```
**be/src/common/status.h:630:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/io/cache/fs_file_cache_storage.cpp:293:** +1, including nesting
penalty of 0, nesting level increased to 1
```cpp
if (USE_CACHE_VERSION2 && version != "2.0") {
^
```
**be/src/io/cache/fs_file_cache_storage.cpp:293:** +1
```cpp
if (USE_CACHE_VERSION2 && version != "2.0") {
^
```
**be/src/io/cache/fs_file_cache_storage.cpp:297:** +2, including nesting
penalty of 1, nesting level increased to 2
```cpp
if (ec) {
^
```
**be/src/io/cache/fs_file_cache_storage.cpp:301:** +2, including nesting
penalty of 1, nesting level increased to 2
```cpp
for (; key_it != std::filesystem::directory_iterator(); ++key_it) {
^
```
**be/src/io/cache/fs_file_cache_storage.cpp:302:** +3, including nesting
penalty of 2, nesting level increased to 3
```cpp
if (key_it->is_directory()) {
^
```
**be/src/io/cache/fs_file_cache_storage.cpp:304:** +4, including nesting
penalty of 3, nesting level increased to 4
```cpp
if (cache_key.size() > KEY_PREFIX_LENGTH) {
^
```
**be/src/io/cache/fs_file_cache_storage.cpp:308:** +5, including nesting
penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(fs->exists(key_prefix, &exists));
^
```
**be/src/common/status.h:628:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do {\
^
```
**be/src/io/cache/fs_file_cache_storage.cpp:308:** +6, including nesting
penalty of 5, nesting level increased to 6
```cpp
RETURN_IF_ERROR(fs->exists(key_prefix, &exists));
^
```
**be/src/common/status.h:630:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/io/cache/fs_file_cache_storage.cpp:309:** +5, including nesting
penalty of 4, nesting level increased to 5
```cpp
if (!exists) {
^
```
**be/src/io/cache/fs_file_cache_storage.cpp:310:** +6, including nesting
penalty of 5, nesting level increased to 6
```cpp
RETURN_IF_ERROR(fs->create_directory(key_prefix));
^
```
**be/src/common/status.h:628:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do {\
^
```
**be/src/io/cache/fs_file_cache_storage.cpp:310:** +7, including nesting
penalty of 6, nesting level increased to 7
```cpp
RETURN_IF_ERROR(fs->create_directory(key_prefix));
^
```
**be/src/common/status.h:630:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/io/cache/fs_file_cache_storage.cpp:312:** +5, including nesting
penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(fs->rename(key_it->path(), key_prefix /
cache_key));
^
```
**be/src/common/status.h:628:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do {\
^
```
**be/src/io/cache/fs_file_cache_storage.cpp:312:** +6, including nesting
penalty of 5, nesting level increased to 6
```cpp
RETURN_IF_ERROR(fs->rename(key_it->path(), key_prefix /
cache_key));
Re: [PR] [fix](OrcWriter) fix be core when upgrading BE without upgrading FE [doris]
BePPPower commented on PR #40303: URL: https://github.com/apache/doris/pull/40303#issuecomment-2325751318 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve]Transform the End-To-End(E2E) tasks on the assembly line [doris-flink-connector]
JNSimba commented on code in PR #466:
URL:
https://github.com/apache/doris-flink-connector/pull/466#discussion_r1741533809
##
flink-doris-connector/src/test/java/org/apache/doris/flink/container/instance/MySQLContainer.java:
##
@@ -0,0 +1,106 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.container.instance;
+
+import org.apache.doris.flink.exception.DorisRuntimeException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testcontainers.lifecycle.Startables;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.SQLException;
+import java.util.stream.Stream;
+
+public class MySQLContainer implements ContainerService {
+private static final Logger LOG =
LoggerFactory.getLogger(MySQLContainer.class);
+private static final String MYSQL_VERSION = "mysql:8.0";
+private static final String USERNAME = "root";
+private static final String PASSWORD = "123456";
+private final org.testcontainers.containers.MySQLContainer mysqlcontainer;
+
+public MySQLContainer() {
+mysqlcontainer = createContainer();
+}
+
+private org.testcontainers.containers.MySQLContainer createContainer() {
+LOG.info("Will create mysql container.");
+return new org.testcontainers.containers.MySQLContainer(MYSQL_VERSION)
+.withUsername(USERNAME)
+.withPassword(PASSWORD);
+}
+
+@Override
+public void startContainer() {
+LOG.info("Starting MySQL container.");
+Startables.deepStart(Stream.of(mysqlcontainer)).join();
+LOG.info("MySQL Container was started.");
+try {
+Thread.sleep(2);
+} catch (InterruptedException e) {
+throw new DorisRuntimeException(e);
+}
Review Comment:
Should this be deleted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](DCHECK) Fix the DCHECK failure when the query is canceled. [doris]
github-actions[bot] commented on PR #40276: URL: https://github.com/apache/doris/pull/40276#issuecomment-2325756031 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](fe) avoid setting the value of batch_size excessively high (#35941) [doris]
doris-robot commented on PR #40312: URL: https://github.com/apache/doris/pull/40312#issuecomment-2325759089 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](tablet report) tablet report increase report version [doris]
yujun777 commented on PR #40172: URL: https://github.com/apache/doris/pull/40172#issuecomment-2325758820 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](fe) avoid setting the value of batch_size excessively high (#35941) [doris]
mrhhsg opened a new pull request, #40312: URL: https://github.com/apache/doris/pull/40312 Issue Number: close #xxx pick #35941 ## Proposed changes Issue Number: close #xxx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](fe) avoid setting the value of batch_size excessively high (#35941) [doris]
mrhhsg commented on PR #40312: URL: https://github.com/apache/doris/pull/40312#issuecomment-2325759625 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](regression test) mv test_report_version_missing to p1 [doris]
yujun777 commented on PR #40184: URL: https://github.com/apache/doris/pull/40184#issuecomment-2325760253 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](inverted index) fix case for index_compaction [doris]
doris-robot commented on PR #40313: URL: https://github.com/apache/doris/pull/40313#issuecomment-2325760760 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](inverted index) fix case for index_compaction [doris]
airborne12 commented on PR #40313: URL: https://github.com/apache/doris/pull/40313#issuecomment-2325761327 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [test](inverted index) fix case for index_compaction [doris]
airborne12 opened a new pull request, #40313: URL: https://github.com/apache/doris/pull/40313 ## Proposed changes Fix unstable case of index compaction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](tablet report) tablet report increase report version [doris]
yujun777 commented on PR #40172: URL: https://github.com/apache/doris/pull/40172#issuecomment-2325766442 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](tablet report) tablet report increase report version [doris]
github-actions[bot] commented on PR #40172: URL: https://github.com/apache/doris/pull/40172#issuecomment-2325769586 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/09: [fix](Nereids) datetimev2 literal equals should compare microsecond (#40121)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ef8397fdc2914166e5ac1b54ddcf776736f2d234
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Fri Aug 30 13:56:05 2024 +0800
[fix](Nereids) datetimev2 literal equals should compare microsecond (#40121)
---
.../nereids/trees/expressions/literal/DateTimeV2Literal.java | 7 ++-
.../nereids/trees/expressions/literal/DateTimeLiteralTest.java | 10 ++
2 files changed, 16 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeV2Literal.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeV2Literal.java
index a3457f2463d..0ca19bf2a92 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeV2Literal.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeV2Literal.java
@@ -104,6 +104,11 @@ public class DateTimeV2Literal extends DateTimeLiteral {
getDataType().toCatalogDataType());
}
+@Override
+public double getDouble() {
+return super.getDouble() + microSecond / 100.0;
+}
+
@Override
public String toString() {
return getStringValue();
@@ -291,6 +296,6 @@ public class DateTimeV2Literal extends DateTimeLiteral {
return false;
}
DateTimeV2Literal literal = (DateTimeV2Literal) o;
-return Objects.equals(dataType, literal.dataType);
+return Objects.equals(dataType, literal.dataType) &&
Objects.equals(microSecond, literal.microSecond);
}
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
index 230a38a..914246f08b6 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteralTest.java
@@ -490,4 +490,14 @@ class DateTimeLiteralTest {
Assertions.assertEquals(1, literal.roundCeiling(0).month);
Assertions.assertEquals(2001, literal.roundCeiling(0).year);
}
+
+@Test
+void testEquals() {
+DateTimeV2Literal l1 = new DateTimeV2Literal(1, 1, 1, 1, 1, 1, 1);
+DateTimeV2Literal l2 = new DateTimeV2Literal(1, 1, 1, 1, 1, 1, 1);
+DateTimeV2Literal l3 = new DateTimeV2Literal(1, 1, 1, 1, 1, 1, 2);
+
+Assertions.assertEquals(l1, l2);
+Assertions.assertNotEquals(l1, l3);
+}
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 02/09: [fix](table-func) fix explode_map with alias (#39757)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 272964412459f698d08344abb6e24d6e07141f2f
Author: amory
AuthorDate: Tue Aug 27 14:42:42 2024 +0800
[fix](table-func) fix explode_map with alias (#39757)
## Proposed changes
when we use sql
```
select
a.name,
tmp.k,
tmp.v
from (
select
'zhangsan' name,
map(1,20,2,30) score
)a lateral view explode_map(score) tmp as k,v;
errCode = 2, detailMessage = Unknown table 'v'
```
after this pr:
```
mysql> select a.name, tmp.k, tmp.v from ( select
'zhangsan' name, map(1,20,2,30) score )a lateral view
explode_map(score) tmp as k,v;
+--+--+--+
| name | k| v|
+--+--+--+
| zhangsan |1 | 20 |
| zhangsan |2 | 30 |
+--+--+--+
2 rows in set (0.31 sec)
```
Issue Number: close #xxx
---
.../nereids/rules/analysis/BindExpression.java | 3 +-
.../doris/nereids/trees/expressions/Alias.java | 4 ++
.../sql_functions/table_function/explode_map.out | 43 ++
.../table_function/explode_map.groovy | 6 +++
4 files changed, 55 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
index db39d661d3e..8f3345a35c0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindExpression.java
@@ -272,7 +272,8 @@ public class BindExpression implements AnalysisRuleFactory {
for (int idx = 0; idx < fields.size(); ++idx) {
expandAlias.add(new Alias(new StructElement(
boundSlot, new
StringLiteral(fields.get(idx).getName())),
-generate.getExpandColumnAlias().get(i).get(idx)));
+generate.getExpandColumnAlias().get(i).get(idx),
+slot.getQualifier()));
}
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Alias.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Alias.java
index 6b31fc60f27..9eea3afd879 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Alias.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Alias.java
@@ -60,6 +60,10 @@ public class Alias extends NamedExpression implements
UnaryExpression {
this(exprId, ImmutableList.of(child), name, ImmutableList.of(), false);
}
+public Alias(Expression child, String name, List qualifier) {
+this(StatementScopeIdGenerator.newExprId(), ImmutableList.of(child),
name, qualifier, false);
+}
+
public Alias(ExprId exprId, Expression child, String name, boolean
nameFromChild) {
this(exprId, ImmutableList.of(child), name, ImmutableList.of(),
nameFromChild);
}
diff --git
a/regression-test/data/nereids_p0/sql_functions/table_function/explode_map.out
b/regression-test/data/nereids_p0/sql_functions/table_function/explode_map.out
index 295b908d7f8..44fb65a8d0c 100644
---
a/regression-test/data/nereids_p0/sql_functions/table_function/explode_map.out
+++
b/regression-test/data/nereids_p0/sql_functions/table_function/explode_map.out
@@ -49,3 +49,46 @@ zhangsan Math60 Chinese 80
zhangsan Math60 English 90
zhangsan Math60 Math60
+-- !explode_sql_alias --
+lisi null\N
+lisi2 \N \N
+wangwu Chinese 88
+wangwu English 96
+wangwu Math90
+zhangsan Chinese 80
+zhangsan English 90
+zhangsan Math60
+
+-- !explode_outer_sql_alias --
+amory \N \N
+lisi null\N
+lisi2 \N \N
+wangwu Chinese 88
+wangwu English 96
+wangwu Math90
+zhangsan Chinese 80
+zhangsan English 90
+zhangsan Math60
+
+-- !explode_sql_alias_multi --
+lisi null\N null\N
+lisi2 \N \N \N \N
+wangwu Chinese 88 Chinese 88
+wangwu Chinese 88 English 96
+wangwu Chinese 88 Math90
+wangwu English 96 Chinese 88
+wangwu English 96 English 96
+wangwu English 96 Math90
+wangwu Math90 Chinese 88
+wangwu Math90 English 96
+wangwu Math90 Math90
+zhangsan Chinese 80 Chinese 80
+zhangsan Chinese 80 English 90
+zhangsan Chinese 80 Math60
+zhangsan English 90 Chinese 80
+zhangsan English 90 English 90
+zhangsan English 90 Math60
+zhangsan Math6
(doris) 07/09: [fix](mtmv) Fix result wrong when query rewrite by mv if query contains null_unsafe equals expression (#39629)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 4fe8b64c2d1e9c78fcfd688a8583b39fc53281c6
Author: seawinde <149132972+seawi...@users.noreply.github.com>
AuthorDate: Wed Aug 28 10:58:08 2024 +0800
[fix](mtmv) Fix result wrong when query rewrite by mv if query contains
null_unsafe equals expression (#39629)
Fix result wrong when query rewrite by mv if query contains null_unsafe
equals expression and the expression both side is slot
table orders data is as following:
(null, 1, 'o', 9.5, '2023-12-08', 'a', 'b', 1, 'yy'),
(1, null, 'o', 10.5, '2023-12-08', 'a', 'b', 1, 'yy'),
(2, 1, null, 11.5, '2023-12-09', 'a', 'b', 1, 'yy'),
(3, 1, 'o', null, '2023-12-10', 'a', 'b', 1, 'yy'),
(3, 1, 'o', 33.5, null, 'a', 'b', 1, 'yy'),
(4, 2, 'o', 43.2, '2023-12-11', null,'d',2, 'mm'),
(5, 2, 'o', 56.2, '2023-12-12', 'c',null, 2, 'mi'),
(5, 2, 'o', 1.2, '2023-12-12', 'c','d', null, 'mi');
such as mv def is
select count(*), o_orderstatus, o_comment
from orders
group by
o_orderstatus, o_comment;
query is as following:
select count(*), o_orderstatus, o_comment
from orders
where o_orderstatus = o_orderstatus
group by
o_orderstatus, o_comment;
after rewrite by materialized view, the result is wrong as following,
the row contains null should not appear
+--+---+---+
| count(*) | o_orderstatus | o_comment |
+--+---+---+
|1 | NULL | yy|
|1 | o | mm|
|2 | o | mi|
|4 | o | yy|
+--+---+---+
---
.../rules/exploration/mv/EquivalenceClass.java | 41 ++-
.../nereids/rules/exploration/mv/Predicates.java | 13 ++--
...etMapping.java => EquivalenceClassMapping.java} | 33 +
.../mv/unsafe_equals/null_un_safe_equals.out | 11 +++
.../mv/unsafe_equals/null_un_safe_equals.groovy| 85 ++
5 files changed, 142 insertions(+), 41 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/EquivalenceClass.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/EquivalenceClass.java
index d6b59b6866c..75a03fade29 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/EquivalenceClass.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/EquivalenceClass.java
@@ -21,12 +21,9 @@ import
org.apache.doris.nereids.trees.expressions.SlotReference;
import java.util.ArrayList;
import java.util.HashMap;
-import java.util.HashSet;
import java.util.LinkedHashMap;
-import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
-import java.util.Set;
/**
* EquivalenceClass, this is used for equality propagation when predicate
compensation
@@ -40,14 +37,19 @@ public class EquivalenceClass {
* a: [a, b],
* b: [a, b]
* }
+ * or column a = a,
+ * this would be
+ * {
+ * a: [a, a]
+ * }
*/
-private Map> equivalenceSlotMap = new
LinkedHashMap<>();
-private List> equivalenceSlotList;
+private Map> equivalenceSlotMap = new
LinkedHashMap<>();
+private List> equivalenceSlotList;
public EquivalenceClass() {
}
-public EquivalenceClass(Map>
equivalenceSlotMap) {
+public EquivalenceClass(Map>
equivalenceSlotMap) {
this.equivalenceSlotMap = equivalenceSlotMap;
}
@@ -56,13 +58,13 @@ public class EquivalenceClass {
*/
public void addEquivalenceClass(SlotReference leftSlot, SlotReference
rightSlot) {
-Set leftSlotSet = equivalenceSlotMap.get(leftSlot);
-Set rightSlotSet = equivalenceSlotMap.get(rightSlot);
+List leftSlotSet = equivalenceSlotMap.get(leftSlot);
+List rightSlotSet = equivalenceSlotMap.get(rightSlot);
if (leftSlotSet != null && rightSlotSet != null) {
// Both present, we need to merge
if (leftSlotSet.size() < rightSlotSet.size()) {
// We swap them to merge
-Set tmp = rightSlotSet;
+List tmp = rightSlotSet;
rightSlotSet = leftSlotSet;
leftSlotSet = tmp;
}
@@ -80,7 +82,7 @@ public class EquivalenceClass {
equivalenceSlotMap.put(leftSlot, rightSlotSet);
} else {
// None are present, add to same equivalence class
-Set equivalenceClass = new LinkedHashSet<>();
+List equivalenceClass = new ArrayList<>();
equivalence
(doris) branch branch-3.0 updated (8bdbd0da564 -> ca4bd831b69)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
from 8bdbd0da564 [Improvement](bloom filter) Forbid small bloom filter
(#38349) (#39387) (#40256)
new ef8397fdc29 [fix](Nereids) datetimev2 literal equals should compare
microsecond (#40121)
new 27296441245 [fix](table-func) fix explode_map with alias (#39757)
new a4a14cb345a [fix](compaction) fix heap-use-after-free caused by
compaction sample info resize (#40058)
new bc7fa2f045b [fix](mtmv) Add debug log decide for performance when
query rewrite by materialized view (#39914)
new 9ce16c77019 [fix](nereids)change lag/lead function signature (#40060)
new 21f557a1efe [fix](funcs) map struct construct funcs (#39699)
new 4fe8b64c2d1 [fix](mtmv) Fix result wrong when query rewrite by mv if
query contains null_unsafe equals expression (#39629)
new d1cf33f2cae [fix](coordinator) fix union all instance number (#3)
new ca4bd831b69 [fix](cloud) Fix migrate tablets between backends back and
forth (#39792)
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/cloud/cloud_backend_service.cpp | 5 +
be/src/olap/merger.cpp | 15 ++-
be/src/vec/functions/function.h| 1 +
be/src/vec/functions/function_map.cpp | 12 +-
be/src/vec/functions/function_struct.cpp | 7 +-
.../doris/cloud/catalog/CloudTabletRebalancer.java | 95 +-
.../nereids/rules/analysis/BindExpression.java | 3 +-
.../mv/AbstractMaterializedViewRule.java | 30 +++--
.../rules/exploration/mv/EquivalenceClass.java | 41 +++---
.../nereids/rules/exploration/mv/Predicates.java | 13 +-
...etMapping.java => EquivalenceClassMapping.java} | 33 ++---
.../doris/nereids/trees/expressions/Alias.java | 4 +
.../functions/window/RequireTrivialTypes.java | 2 -
.../expressions/literal/DateTimeV2Literal.java | 7 +-
.../main/java/org/apache/doris/qe/Coordinator.java | 43 ---
.../expressions/literal/DateTimeLiteralTest.java | 10 ++
.../query/map_functions/test_map_with_agg.out | 9 ++
.../sql_functions/table_function/explode_map.out | 43 +++
.../mv/unsafe_equals/null_un_safe_equals.out | 11 ++
.../multi_cluster/test_warmup_rebalance.groovy | 137 +
.../query/map_functions/test_map_with_agg.groovy | 55 +
.../table_function/explode_map.groovy | 6 +
.../mv/unsafe_equals/null_un_safe_equals.groovy| 85 +
.../nereids_syntax_p0/lag_lead_signature.groovy| 35 ++
.../query_p0/union/test_union_instance.groovy | 88 +
25 files changed, 669 insertions(+), 121 deletions(-)
rename
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/mapping/{EquivalenceClassSetMapping.java
=> EquivalenceClassMapping.java} (54%)
create mode 100644
regression-test/data/datatype_p0/nested_types/query/map_functions/test_map_with_agg.out
create mode 100644
regression-test/data/nereids_rules_p0/mv/unsafe_equals/null_un_safe_equals.out
create mode 100644
regression-test/suites/cloud_p0/multi_cluster/test_warmup_rebalance.groovy
create mode 100644
regression-test/suites/datatype_p0/nested_types/query/map_functions/test_map_with_agg.groovy
create mode 100644
regression-test/suites/nereids_rules_p0/mv/unsafe_equals/null_un_safe_equals.groovy
create mode 100644
regression-test/suites/nereids_syntax_p0/lag_lead_signature.groovy
create mode 100644
regression-test/suites/query_p0/union/test_union_instance.groovy
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 05/09: [fix](nereids)change lag/lead function signature (#40060)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 9ce16c7701912bea9ea762bbd7f9933b6b6e3960
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Fri Aug 30 10:56:25 2024 +0800
[fix](nereids)change lag/lead function signature (#40060)
remove CharType from the signature
---
.../functions/window/RequireTrivialTypes.java | 2 --
.../nereids_syntax_p0/lag_lead_signature.groovy| 35 ++
2 files changed, 35 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/RequireTrivialTypes.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/RequireTrivialTypes.java
index 34295732d58..04d92fdeb11 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/RequireTrivialTypes.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/window/RequireTrivialTypes.java
@@ -19,7 +19,6 @@ package
org.apache.doris.nereids.trees.expressions.functions.window;
import org.apache.doris.nereids.types.BigIntType;
import org.apache.doris.nereids.types.BooleanType;
-import org.apache.doris.nereids.types.CharType;
import org.apache.doris.nereids.types.DataType;
import org.apache.doris.nereids.types.DateTimeType;
import org.apache.doris.nereids.types.DateTimeV2Type;
@@ -64,7 +63,6 @@ public interface RequireTrivialTypes {
DateTimeV2Type.SYSTEM_DEFAULT,
TimeType.INSTANCE,
TimeV2Type.INSTANCE,
-CharType.SYSTEM_DEFAULT,
VarcharType.SYSTEM_DEFAULT,
StringType.INSTANCE
);
diff --git a/regression-test/suites/nereids_syntax_p0/lag_lead_signature.groovy
b/regression-test/suites/nereids_syntax_p0/lag_lead_signature.groovy
new file mode 100644
index 000..5dde98a95a0
--- /dev/null
+++ b/regression-test/suites/nereids_syntax_p0/lag_lead_signature.groovy
@@ -0,0 +1,35 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("lag_lead_signature") {
+sql "SET enable_nereids_planner=true"
+sql "SET enable_fallback_to_original_planner=false"
+sql """drop table if exists lag_lead_signature_t"""
+sql """ CREATE TABLE lag_lead_signature_t (
+`k` VARCHAR(200) NULL
+) ENGINE=OLAP
+DUPLICATE KEY(`k`)
+DISTRIBUTED BY HASH(`k`) BUCKETS 3
+PROPERTIES (
+"replication_allocation" = "tag.location.default: 1"
+);"""
+
+sql "insert into lag_lead_signature_t values ('44'), ('28');"
+
+sql "select lag(k, 1, 0) over(), lead(k, 1, 0) over() from
lag_lead_signature_t;"
+
+}
\ No newline at end of file
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 08/09: [fix](coordinator) fix union all instance number (#39999)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit d1cf33f2cae735733924bfded9875b820af5f5b7
Author: xzj7019 <13794+xzj7...@users.noreply.github.com>
AuthorDate: Thu Aug 29 11:26:51 2024 +0800
[fix](coordinator) fix union all instance number (#3)
In following plan pattern, fragment 11 has 3 child fragments aligning
with 3 exchange nodes and the union plan node only have 2 plan children,
i.e, the aggr node and the exchange node(4). In current union node's
parallelism computing, it used the plan children's size to do iteration
to find the max parallel but ignore the last fragment's. Actually, it
should use fragment's children(exclude unpartitioned ones) instead of
plan node's children to do the iteration.
11:VUNION(703)
|
|10:VAGGREGATE (update finalize)(697)
|| group by: brand_name[#23], skc[#24]
|| sortByGroupKey:false
|| cardinality=1
|| final projections: '1', brand_name[#25], skc[#26]
|| final project output tuple id: 11
|| distribute expr lists: skc[#24]
||
|9:VHASH JOIN(691)
|| join op: INNER JOIN(PARTITIONED)[]
|| equal join conjunct: (skc[#19] = skc_code[#12])
|| cardinality=1
|| vec output tuple id: 9
|| output tuple id: 9
|| vIntermediate tuple ids: 8
|| hash output slot ids: 18 19
|| final projections: brand_name[#20], skc[#21]
|| final project output tuple id: 9
|| distribute expr lists: skc[#19]
|| distribute expr lists: skc_code[#12]
||
||6:VEXCHANGE
|| offset: 0
|| distribute expr lists: skc_code[#12]
||
|8:VEXCHANGE
| offset: 0
| distribute expr lists:
|
4:VEXCHANGE
offset: 0
distribute expr lists:
---
.../main/java/org/apache/doris/qe/Coordinator.java | 43 ++-
.../query_p0/union/test_union_instance.groovy | 88 ++
2 files changed, 111 insertions(+), 20 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
index 69311d3a029..ef5c3c3e4f1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
@@ -1787,7 +1787,6 @@ public class Coordinator implements CoordInterface {
}
Pair pairNodes =
findLeftmostNode(fragment.getPlanRoot());
-PlanNode fatherNode = pairNodes.first;
PlanNode leftMostNode = pairNodes.second;
/*
@@ -1802,25 +1801,8 @@ public class Coordinator implements CoordInterface {
// (Case B)
// there is no leftmost scan; we assign the same hosts as
those of our
// input fragment which has a higher instance_number
-
-int inputFragmentIndex = 0;
-int maxParallelism = 0;
-// If the fragment has three children, then the first child
and the second child are
-// the children(both exchange node) of shuffle HashJoinNode,
-// and the third child is the right child(ExchangeNode) of
broadcast HashJoinNode.
-// We only need to pay attention to the maximum parallelism
among
-// the two ExchangeNodes of shuffle HashJoinNode.
-int childrenCount = (fatherNode != null) ?
fatherNode.getChildren().size() : 1;
-for (int j = 0; j < childrenCount; j++) {
-int currentChildFragmentParallelism
-=
fragmentExecParamsMap.get(fragment.getChild(j).getFragmentId()).instanceExecParams.size();
-if (currentChildFragmentParallelism > maxParallelism) {
-maxParallelism = currentChildFragmentParallelism;
-inputFragmentIndex = j;
-}
-}
-
-PlanFragmentId inputFragmentId =
fragment.getChild(inputFragmentIndex).getFragmentId();
+int maxParallelFragmentIndex =
findMaxParallelFragmentIndex(fragment);
+PlanFragmentId inputFragmentId =
fragment.getChild(maxParallelFragmentIndex).getFragmentId();
// AddAll() soft copy()
int exchangeInstances = -1;
if (ConnectContext.get() != null &&
ConnectContext.get().getSessionVariable() != null) {
@@ -1975,6 +1957,27 @@ public class Coordinator implements CoordInterface {
}
}
+private int findMaxParallelFragmentIndex(PlanFragment fragment) {
+Preconditions.checkState(!fragment.getChildren().isEmpty(), "fr
(doris) 09/09: [fix](cloud) Fix migrate tablets between backends back and forth (#39792)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ca4bd831b69d7a13d1fd5f6289cdca5460e9b981
Author: deardeng <565620...@qq.com>
AuthorDate: Mon Sep 2 00:29:06 2024 +0800
[fix](cloud) Fix migrate tablets between backends back and forth (#39792)
BUG: cloud rebalancer migrates tablets back and forth: move from A to B,
then B to A, then A to B, ...
The reason is that the tabletToInfightTask map tracking in-flight tasks
ignored the multi-cluster scenario, and in the statRouteInfo function,
the cluster information was lost, which led to inaccurate tablets
statistics.
---
be/src/cloud/cloud_backend_service.cpp | 5 +
.../doris/cloud/catalog/CloudTabletRebalancer.java | 95 +-
.../multi_cluster/test_warmup_rebalance.groovy | 137 +
3 files changed, 205 insertions(+), 32 deletions(-)
diff --git a/be/src/cloud/cloud_backend_service.cpp
b/be/src/cloud/cloud_backend_service.cpp
index d91e9e416b8..2dc6d03ebf6 100644
--- a/be/src/cloud/cloud_backend_service.cpp
+++ b/be/src/cloud/cloud_backend_service.cpp
@@ -180,6 +180,11 @@ void
CloudBackendService::check_warm_up_cache_async(TCheckWarmUpCacheAsyncRespon
const
TCheckWarmUpCacheAsyncRequest& request) {
std::map task_done;
_engine.file_cache_block_downloader().check_download_task(request.tablets,
&task_done);
+
DBUG_EXECUTE_IF("CloudBackendService.check_warm_up_cache_async.return_task_false",
{
+for (auto& it : task_done) {
+it.second = false;
+}
+});
response.__set_task_done(task_done);
Status st = Status::OK();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletRebalancer.java
b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletRebalancer.java
index 73ddbe4c455..fc580c4fc7e 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletRebalancer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/cloud/catalog/CloudTabletRebalancer.java
@@ -46,6 +46,7 @@ import org.apache.doris.thrift.TWarmUpCacheAsyncRequest;
import org.apache.doris.thrift.TWarmUpCacheAsyncResponse;
import com.google.common.base.Preconditions;
+import lombok.Getter;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -54,6 +55,7 @@ import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
+import java.util.Objects;
import java.util.Random;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
@@ -94,7 +96,7 @@ public class CloudTabletRebalancer extends MasterDaemon {
private LinkedBlockingQueue> tabletsMigrateTasks = new
LinkedBlockingQueue>();
-private Map tabletToInfightTask = new HashMap();
+private Map tabletToInfightTask = new
HashMap<>();
private long assignedErrNum = 0;
@@ -115,12 +117,39 @@ public class CloudTabletRebalancer extends MasterDaemon {
PARTITION
}
+@Getter
+private class InfightTablet {
+private final Long tabletId;
+private final String clusterId;
+
+public InfightTablet(Long tabletId, String clusterId) {
+this.tabletId = tabletId;
+this.clusterId = clusterId;
+}
+
+@Override
+public boolean equals(Object o) {
+if (this == o) {
+return true;
+}
+if (o == null || getClass() != o.getClass()) {
+return false;
+}
+InfightTablet that = (InfightTablet) o;
+return tabletId.equals(that.tabletId) &&
clusterId.equals(that.clusterId);
+}
+
+@Override
+public int hashCode() {
+return Objects.hash(tabletId, clusterId);
+}
+}
+
private class InfightTask {
public Tablet pickedTablet;
public long srcBe;
public long destBe;
public boolean isGlobal;
-public String clusterId;
public Map> beToTablets;
public long startTimestamp;
BalanceType balanceType;
@@ -343,41 +372,44 @@ public class CloudTabletRebalancer extends MasterDaemon {
}
public void checkInflghtWarmUpCacheAsync() {
-Map> beToTabletIds = new HashMap>();
+Map> beToInfightTasks = new HashMap>();
-for (Map.Entry entry :
tabletToInfightTask.entrySet()) {
-beToTabletIds.putIfAbsent(entry.getValue().destBe, new
ArrayList());
-
beToTabletIds.get(entry.getValue().destBe).add(entry.getValue().pickedTablet.getId());
+for (Map.Entry entry :
tabletToInfightTask.entrySet()) {
+beToInfightTasks.putIfAbsent(entry.getValue().destBe, new
ArrayList<>());
+
beToInfightTasks.get(entry.getValue().destBe).add(en
(doris) 03/09: [fix](compaction) fix heap-use-after-free caused by compaction sample info resize (#40058)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit a4a14cb345ab7b5a681d2c1cd9ddb4d477787177
Author: Luwei <814383...@qq.com>
AuthorDate: Wed Aug 28 23:16:37 2024 +0800
[fix](compaction) fix heap-use-after-free caused by compaction sample info
resize (#40058)
---
be/src/olap/merger.cpp | 15 ---
1 file changed, 12 insertions(+), 3 deletions(-)
diff --git a/be/src/olap/merger.cpp b/be/src/olap/merger.cpp
index 87792db93a6..1bba7304638 100644
--- a/be/src/olap/merger.cpp
+++ b/be/src/olap/merger.cpp
@@ -459,7 +459,10 @@ Status Merger::vertical_merge_rowsets(BaseTabletSPtr
tablet, ReaderType reader_t
vectorized::RowSourcesBuffer row_sources_buf(
tablet->tablet_id(), dst_rowset_writer->context().tablet_path,
reader_type);
-tablet->sample_infos.resize(column_groups.size(), {0, 0, 0});
+{
+std::unique_lock lock(tablet->sample_info_lock);
+tablet->sample_infos.resize(column_groups.size(), {0, 0, 0});
+}
// compact group one by one
for (auto i = 0; i < column_groups.size(); ++i) {
VLOG_NOTICE << "row source size: " << row_sources_buf.total_size();
@@ -467,10 +470,16 @@ Status Merger::vertical_merge_rowsets(BaseTabletSPtr
tablet, ReaderType reader_t
int64_t batch_size = config::compaction_batch_size != -1
? config::compaction_batch_size
: estimate_batch_size(i, tablet,
merge_way_num);
-RETURN_IF_ERROR(vertical_compact_one_group(
+CompactionSampleInfo sample_info;
+Status st = vertical_compact_one_group(
tablet, reader_type, tablet_schema, is_key, column_groups[i],
&row_sources_buf,
src_rowset_readers, dst_rowset_writer, max_rows_per_segment,
stats_output,
-key_group_cluster_key_idxes, batch_size,
&(tablet->sample_infos[i])));
+key_group_cluster_key_idxes, batch_size, &sample_info);
+{
+std::unique_lock lock(tablet->sample_info_lock);
+tablet->sample_infos[i] = sample_info;
+}
+RETURN_IF_ERROR(st);
if (is_key) {
RETURN_IF_ERROR(row_sources_buf.flush());
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 06/09: [fix](funcs) map struct construct funcs (#39699)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 21f557a1efebdb1f3abce5bc703e49bb68d97660
Author: amory
AuthorDate: Tue Aug 27 14:55:17 2024 +0800
[fix](funcs) map struct construct funcs (#39699)
we should not change the column which is passed from a block when we
execute a function , because this block may be used in other expression
or other exec node
---
be/src/vec/functions/function.h| 1 +
be/src/vec/functions/function_map.cpp | 12 ++---
be/src/vec/functions/function_struct.cpp | 7 +--
.../query/map_functions/test_map_with_agg.out | 9
.../query/map_functions/test_map_with_agg.groovy | 55 ++
5 files changed, 75 insertions(+), 9 deletions(-)
diff --git a/be/src/vec/functions/function.h b/be/src/vec/functions/function.h
index a77812559b0..43ab3aaba92 100644
--- a/be/src/vec/functions/function.h
+++ b/be/src/vec/functions/function.h
@@ -388,6 +388,7 @@ class IFunction : public
std::enable_shared_from_this,
public:
String get_name() const override = 0;
+/// Notice: We should not change the column in the block, because the
column may be shared by multiple expressions or exec nodes.
virtual Status execute_impl(FunctionContext* context, Block& block,
const ColumnNumbers& arguments, size_t result,
size_t input_rows_count) const override = 0;
diff --git a/be/src/vec/functions/function_map.cpp
b/be/src/vec/functions/function_map.cpp
index 8e8f6b28cd8..3269dde41cb 100644
--- a/be/src/vec/functions/function_map.cpp
+++ b/be/src/vec/functions/function_map.cpp
@@ -104,14 +104,16 @@ public:
result_col_map_offsets.resize(input_rows_count);
std::unique_ptr col_const =
std::make_unique_for_overwrite(num_element);
+std::vector arg(num_element);
for (size_t i = 0; i < num_element; ++i) {
auto& col = block.get_by_position(arguments[i]).column;
std::tie(col, col_const[i]) = unpack_if_const(col);
bool is_nullable = i % 2 == 0 ?
result_col_map_keys_data.is_nullable()
:
result_col_map_vals_data.is_nullable();
// convert to nullable column
+arg[i] = col;
if (is_nullable && !col->is_nullable()) {
-col = ColumnNullable::create(col,
ColumnUInt8::create(col->size(), 0));
+arg[i] = ColumnNullable::create(col,
ColumnUInt8::create(col->size(), 0));
}
}
@@ -119,11 +121,9 @@ public:
ColumnArray::Offset64 offset = 0;
for (size_t row = 0; row < input_rows_count; ++row) {
for (size_t i = 0; i < num_element; i += 2) {
-
result_col_map_keys_data.insert_from(*block.get_by_position(arguments[i]).column,
- index_check_const(row,
col_const[i]));
-result_col_map_vals_data.insert_from(
-*block.get_by_position(arguments[i + 1]).column,
-index_check_const(row, col_const[i + 1]));
+result_col_map_keys_data.insert_from(*arg[i],
index_check_const(row, col_const[i]));
+result_col_map_vals_data.insert_from(*arg[i + 1],
+ index_check_const(row,
col_const[i + 1]));
}
offset += num_element / 2;
result_col_map_offsets[row] = offset;
diff --git a/be/src/vec/functions/function_struct.cpp
b/be/src/vec/functions/function_struct.cpp
index c3a5eb6dc56..a106c14b835 100644
--- a/be/src/vec/functions/function_struct.cpp
+++ b/be/src/vec/functions/function_struct.cpp
@@ -93,21 +93,22 @@ public:
"function {} args number {} is not equal to result struct
field number {}.",
get_name(), num_element, struct_column->tuple_size());
}
+std::vector arg(num_element);
for (size_t i = 0; i < num_element; ++i) {
auto& nested_col = struct_column->get_column(i);
nested_col.reserve(input_rows_count);
bool is_nullable = nested_col.is_nullable();
auto& col = block.get_by_position(args_num[i]).column;
col = col->convert_to_full_column_if_const();
+arg[i] = col;
if (is_nullable && !col->is_nullable()) {
-col = ColumnNullable::create(col,
ColumnUInt8::create(col->size(), 0));
+arg[i] = ColumnNullable::create(col,
ColumnUInt8::create(col->size(), 0));
}
}
// insert value into struct column by column
for (size_t i = 0; i < num_element; ++i) {
-struct_column->get_column(i).insert_range_from(
-*block.get_by_po
(doris) 04/09: [fix](mtmv) Add debug log decide for performance when query rewrite by materialized view (#39914)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit bc7fa2f045b973ae5541b24be1926325b1d40209
Author: seawinde <149132972+seawi...@users.noreply.github.com>
AuthorDate: Tue Aug 27 10:10:49 2024 +0800
[fix](mtmv) Add debug log decide for performance when query rewrite by
materialized view (#39914)
## Proposed changes
In method `AbstractMaterializedViewRule#isMaterializationValid`
Should add `LOG.isDebugEnabled()` before print debug log.
Because `Plan#treeString` in debug log is performance consume.
---
.../mv/AbstractMaterializedViewRule.java | 30 +-
1 file changed, 18 insertions(+), 12 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
index 857fd0e51b9..9fef549c0a7 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
@@ -853,10 +853,12 @@ public abstract class AbstractMaterializedViewRule
implements ExplorationRuleFac
"View struct info is invalid", () ->
String.format("view plan is %s",
context.getStructInfo().getOriginalPlan().treeString()));
// tmp to location question
-LOG.debug(String.format("View struct info is invalid, mv
identifier is %s, query plan is %s,"
-+ "view plan is %s",
-context.generateMaterializationIdentifier(),
queryPlan.treeString(),
-context.getStructInfo().getTopPlan().treeString()));
+if (LOG.isDebugEnabled()) {
+LOG.debug(String.format("View struct info is invalid, mv
identifier is %s, query plan is %s,"
++ "view plan is %s",
+context.generateMaterializationIdentifier(),
queryPlan.treeString(),
+
context.getStructInfo().getTopPlan().treeString()));
+}
cascadesContext.getMemo().recordMaterializationCheckResult(this.getClass(),
materializationId,
false);
return false;
@@ -868,20 +870,24 @@ public abstract class AbstractMaterializedViewRule
implements ExplorationRuleFac
context.recordFailReason(context.getStructInfo(),
"View struct info is invalid", () -> String.format("view
plan is %s",
context.getStructInfo().getOriginalPlan().treeString()));
-LOG.debug(String.format("View struct info is invalid, mv
identifier is %s, query plan is %s,"
-+ "view plan is %s",
-context.generateMaterializationIdentifier(),
queryPlan.treeString(),
-context.getStructInfo().getTopPlan().treeString()));
+if (LOG.isDebugEnabled()) {
+LOG.debug(String.format("View struct info is invalid, mv
identifier is %s, query plan is %s,"
++ "view plan is %s",
+context.generateMaterializationIdentifier(),
queryPlan.treeString(),
+context.getStructInfo().getTopPlan().treeString()));
+}
return false;
}
if (!context.getStructInfo().isValid()) {
context.recordFailReason(context.getStructInfo(),
"View original struct info is invalid", () ->
String.format("view plan is %s",
context.getStructInfo().getOriginalPlan().treeString()));
-LOG.debug(String.format("View struct info is invalid, mv
identifier is %s, query plan is %s,"
-+ "view plan is %s",
-context.generateMaterializationIdentifier(),
queryPlan.treeString(),
-context.getStructInfo().getTopPlan().treeString()));
+if (LOG.isDebugEnabled()) {
+LOG.debug(String.format("View struct info is invalid, mv
identifier is %s, query plan is %s,"
++ "view plan is %s",
+context.generateMaterializationIdentifier(),
queryPlan.treeString(),
+context.getStructInfo().getTopPlan().treeString()));
+}
return false;
}
return true;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](ES Catalog)Do not extract doc_values for field with ignore_above setting [doris]
qidaye opened a new pull request, #40314: URL: https://github.com/apache/doris/pull/40314 ## Proposed changes Issue Number: close #xxx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ES Catalog)Do not extract doc_values for field with ignore_above setting [doris]
doris-robot commented on PR #40314: URL: https://github.com/apache/doris/pull/40314#issuecomment-2325772534 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ES Catalog)Do not extract doc_values for field with ignore_above setting [doris]
qidaye commented on PR #40314: URL: https://github.com/apache/doris/pull/40314#issuecomment-2325772675 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](tablet report) tablet report increase report version [doris]
github-actions[bot] commented on PR #40172: URL: https://github.com/apache/doris/pull/40172#issuecomment-2325776757 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](decompressor) impl ZstdDecompressor [doris]
suxiaogang223 commented on PR #40315: URL: https://github.com/apache/doris/pull/40315#issuecomment-2325778188 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](decompressor) impl ZstdDecompressor [doris]
suxiaogang223 opened a new pull request, #40315: URL: https://github.com/apache/doris/pull/40315 ## Proposed changes Impl ZstdDecompressor and support read hive text table which is compressed by zstd. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](decompressor) impl ZstdDecompressor [doris]
doris-robot commented on PR #40315: URL: https://github.com/apache/doris/pull/40315#issuecomment-2325777888 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] Update lakehouse-overview.md [doris-website]
zhangm365 opened a new pull request, #1082: URL: https://github.com/apache/doris-website/pull/1082 fix the latest dev docs about the lakehouse-overview. # Versions - [x] dev - [ ] 3.0 - [ ] 2.1 - [ ] 2.0 # Languages - [x] Chinese - [ ] English -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhencement](sample) add delta and kudu samples [doris]
github-actions[bot] commented on PR #40067:
URL: https://github.com/apache/doris/pull/40067#issuecomment-2325782373
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/10678376874";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh line
162:
export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
^^ SC2155 (warning): Declare and assign separately to
avoid masking return values.
In samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh line
20:
export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
^^ SC2155 (warning): Declare and assign separately to
avoid masking return values.
For more information:
https://www.shellcheck.net/wiki/SC2155 -- Declare and assign separately to
...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' returned error 1 finding the following formatting issues:
--
--- samples/datalake/deltalake_and_kudu/scripts/start_doris.sh.orig
+++ samples/datalake/deltalake_and_kudu/scripts/start_doris.sh
@@ -20,8 +20,8 @@
export JAVA_HOME=/opt/jdk-17.0.2
cp -r /opt/doris-bin /opt/doris
-echo "trino_connector_plugin_dir=/opt/connectors/" >>
/opt/doris/fe/conf/fe.conf
-echo "trino_connector_plugin_dir=/opt/connectors/" >>
/opt/doris/be/conf/be.conf
+echo "trino_connector_plugin_dir=/opt/connectors/"
>>/opt/doris/fe/conf/fe.conf
+echo "trino_connector_plugin_dir=/opt/connectors/"
>>/opt/doris/be/conf/be.conf
/opt/doris/fe/bin/start_fe.sh --daemon
/opt/doris/be/bin/start_be.sh --daemon
--- samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh.orig
+++ samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh
@@ -33,7 +33,6 @@
hdfs_plugin="ff4a3e3b32dcce27f4df58f17938abde"
kudu_java_example="1afe0a890785e8d0011ea7342ae5e43d"
-
download_source_file() {
local FILE_PATH="$1"
local EXPECTED_MD5="$2"
@@ -79,9 +78,6 @@
download_source_file "trino-hdfs-435-20240724.tar.gz" "${hdfs_plugin}"
"https://github.com/apache/doris-thirdparty/releases/download/trino-435-20240724";
download_source_file "kudu-java-example-1.0-SNAPSHOT.jar"
"${kudu_java_example}"
"https://github.com/apache/doris-thirdparty/releases/download/trino-435-20240724";
-
-
-
if [[ ! -f "jdk-17.0.2/SUCCESS" ]]; then
echo "Prepare jdk17 environment"
if [[ -d "jdk-17.0.2" ]]; then
@@ -156,10 +152,9 @@
touch connectors/trino-delta-lake-435/hdfs/SUCCESS
fi
-
cd ../
-export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
+export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
docker compose -f trinoconnector-compose.yml --env-file
trinoconnector-compose.env up -d
echo "Create hive table ..."
--- samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh.orig
+++ samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh
@@ -17,6 +17,6 @@
# specific language governing permissions and limitations
# under the License.
-export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
+export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
docker compose -f trinoconnector-compose.yml --env-file
trinoconnector-compose.env down
--
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) fix bug for A>n, where A.max is infinity [doris]
englefly commented on PR #39936: URL: https://github.com/apache/doris/pull/39936#issuecomment-2325782770 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](OrcWriter) fix be core when upgrading BE without upgrading FE [doris]
doris-robot commented on PR #40282: URL: https://github.com/apache/doris/pull/40282#issuecomment-2325783004 TPC-H: Total hot run time: 39133 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit ef2f8bc0c89257fae048248898dd61c2f6bd79a7, data reload: false -- Round 1 -- q1 17663 465045064506 q2 2027202 215 202 q3 10416 112711661127 q4 10161 802 759 759 q5 7449113911011101 q6 204 150 152 150 q7 923 539 518 518 q8 9355139014261390 q9 8692690969156909 q10 9785477048504770 q11 460 271 281 271 q12 516 253 252 252 q13 18955 311931003100 q14 484 455 430 430 q15 586 555 542 542 q16 678 583 544 544 q17 995 691 692 691 q18 7540706670267026 q19 1400100910681009 q20 717 371 379 371 q21 3648335731573157 q22 383 314 308 308 Total cold run time: 113037 ms Total hot run time: 39133 ms - Round 2, with runtime_filter_mode=off - q1 4417432243024302 q2 532 444 433 433 q3 2879268226662666 q4 1962169217761692 q5 5668563756845637 q6 230 134 134 134 q7 8698816882038168 q8 3675380139483801 q9 17754 17835 18214 17835 q10 5425516851575157 q11 1325112512061125 q12 864 639 660 639 q13 9988329233033292 q14 462 446 432 432 q15 578 528 540 528 q16 735 685 659 659 q17 3191278327652765 q18 8303791876917691 q19 1757164215901590 q20 2133185018701850 q21 12179 12012 11743 11743 q22 616 527 500 500 Total cold run time: 93371 ms Total hot run time: 82639 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve]Transform the End-To-End(E2E) tasks on the assembly line [doris-flink-connector]
JNSimba commented on code in PR #466:
URL:
https://github.com/apache/doris-flink-connector/pull/466#discussion_r1741558446
##
flink-doris-connector/src/test/java/org/apache/doris/flink/container/AbstractE2EService.java:
##
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.container;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import org.apache.doris.flink.container.e2e.CustomerSingleThreadExecutor;
+import org.apache.doris.flink.container.instance.ContainerService;
+import org.apache.doris.flink.container.instance.MySQLContainer;
+import org.apache.doris.flink.exception.DorisRuntimeException;
+import org.apache.doris.flink.tools.cdc.CdcTools;
+import org.apache.doris.flink.tools.cdc.DatabaseSyncConfig;
+import org.junit.BeforeClass;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.Semaphore;
+
+public abstract class AbstractE2EService extends AbstractContainerTestBase {
+private static final Logger LOG =
LoggerFactory.getLogger(AbstractE2EService.class);
+private static ContainerService mysqlContainerService;
+private static final CustomerSingleThreadExecutor singleThreadExecutor =
+new CustomerSingleThreadExecutor();
+protected static final Semaphore SEMAPHORE = new Semaphore(1);
+protected static final String SINK_CONF = "--" +
DatabaseSyncConfig.SINK_CONF;
+protected static final String DORIS_DATABASE = "--database";
+protected static final String HOSTNAME = "hostname";
+protected static final String PORT = "port";
+protected static final String USERNAME = "username";
+protected static final String PASSWORD = "password";
+protected static final String DATABASE_NAME = "database-name";
+protected static final String FENODES = "fenodes";
+protected static final String JDBC_URL = "jdbc-url";
+protected static final String SINK_LABEL_PREFIX = "sink.label-prefix";
+
+@BeforeClass
+public static void initE2EContainers() {
+LOG.info("Trying to Start init E2E containers.");
+initMySQLContainer();
+}
+
+private static void initMySQLContainer() {
+if (Objects.nonNull(mysqlContainerService) &&
mysqlContainerService.isRunning()) {
+LOG.info("The MySQL container has been started and is running
status.");
+return;
+}
+mysqlContainerService = new MySQLContainer();
+mysqlContainerService.startContainer();
+LOG.info("Mysql container was started.");
+}
+
+protected String getMySQLInstanceHost() {
+return mysqlContainerService.getInstanceHost();
+}
+
+protected Integer getMySQLQueryPort() {
+return mysqlContainerService.getMappedPort(3306);
+}
+
+protected String getMySQLUsername() {
+return mysqlContainerService.getUsername();
+}
+
+protected String getMySQLPassword() {
+return mysqlContainerService.getPassword();
+}
+
+protected Connection getMySQLQueryConnection() {
+return mysqlContainerService.getQueryConnection();
+}
+
+protected void submitE2EJob(String jobName, String[] args) {
+singleThreadExecutor.submitJob(
+jobName,
+() -> {
+try {
+LOG.info("{} e2e job will submit to start.", jobName);
+
CdcTools.setStreamExecutionEnvironmentForTesting(configFlinkEnvironment());
+CdcTools.main(args);
+Thread.sleep(1);
+} catch (Exception e) {
+throw new DorisRuntimeException(e);
+}
+});
+}
+
+protected void cancelCurrentE2EJob(String jobName) {
+LOG.info("{} e2e job will cancel", jobName);
+singleThreadExecutor.cancelCurrentJob(jobName);
+}
Review Comment:
Re: [PR] [improve]Transform the End-To-End(E2E) tasks on the assembly line [doris-flink-connector]
JNSimba commented on code in PR #466:
URL:
https://github.com/apache/doris-flink-connector/pull/466#discussion_r1741559069
##
flink-doris-connector/src/test/java/org/apache/doris/flink/container/AbstractE2EService.java:
##
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.container;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import org.apache.doris.flink.container.e2e.CustomerSingleThreadExecutor;
+import org.apache.doris.flink.container.instance.ContainerService;
+import org.apache.doris.flink.container.instance.MySQLContainer;
+import org.apache.doris.flink.exception.DorisRuntimeException;
+import org.apache.doris.flink.tools.cdc.CdcTools;
+import org.apache.doris.flink.tools.cdc.DatabaseSyncConfig;
+import org.junit.BeforeClass;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.Semaphore;
+
+public abstract class AbstractE2EService extends AbstractContainerTestBase {
+private static final Logger LOG =
LoggerFactory.getLogger(AbstractE2EService.class);
+private static ContainerService mysqlContainerService;
+private static final CustomerSingleThreadExecutor singleThreadExecutor =
+new CustomerSingleThreadExecutor();
+protected static final Semaphore SEMAPHORE = new Semaphore(1);
+protected static final String SINK_CONF = "--" +
DatabaseSyncConfig.SINK_CONF;
+protected static final String DORIS_DATABASE = "--database";
+protected static final String HOSTNAME = "hostname";
+protected static final String PORT = "port";
+protected static final String USERNAME = "username";
+protected static final String PASSWORD = "password";
+protected static final String DATABASE_NAME = "database-name";
+protected static final String FENODES = "fenodes";
+protected static final String JDBC_URL = "jdbc-url";
+protected static final String SINK_LABEL_PREFIX = "sink.label-prefix";
+
+@BeforeClass
+public static void initE2EContainers() {
+LOG.info("Trying to Start init E2E containers.");
+initMySQLContainer();
+}
+
+private static void initMySQLContainer() {
+if (Objects.nonNull(mysqlContainerService) &&
mysqlContainerService.isRunning()) {
+LOG.info("The MySQL container has been started and is running
status.");
+return;
+}
+mysqlContainerService = new MySQLContainer();
+mysqlContainerService.startContainer();
+LOG.info("Mysql container was started.");
+}
+
+protected String getMySQLInstanceHost() {
+return mysqlContainerService.getInstanceHost();
+}
+
+protected Integer getMySQLQueryPort() {
+return mysqlContainerService.getMappedPort(3306);
+}
+
+protected String getMySQLUsername() {
+return mysqlContainerService.getUsername();
+}
+
+protected String getMySQLPassword() {
+return mysqlContainerService.getPassword();
+}
+
+protected Connection getMySQLQueryConnection() {
+return mysqlContainerService.getQueryConnection();
+}
+
+protected void submitE2EJob(String jobName, String[] args) {
+singleThreadExecutor.submitJob(
+jobName,
+() -> {
+try {
+LOG.info("{} e2e job will submit to start.", jobName);
+
CdcTools.setStreamExecutionEnvironmentForTesting(configFlinkEnvironment());
+CdcTools.main(args);
+Thread.sleep(1);
+} catch (Exception e) {
+throw new DorisRuntimeException(e);
+}
+});
+}
+
+protected void cancelCurrentE2EJob(String jobName) {
+LOG.info("{} e2e job will cancel", jobName);
+singleThreadExecutor.cancelCurrentJob(jobName);
+}
+
+private Stre
Re: [PR] [improve]Transform the End-To-End(E2E) tasks on the assembly line [doris-flink-connector]
JNSimba commented on code in PR #466:
URL:
https://github.com/apache/doris-flink-connector/pull/466#discussion_r1741558446
##
flink-doris-connector/src/test/java/org/apache/doris/flink/container/AbstractE2EService.java:
##
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.container;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import org.apache.doris.flink.container.e2e.CustomerSingleThreadExecutor;
+import org.apache.doris.flink.container.instance.ContainerService;
+import org.apache.doris.flink.container.instance.MySQLContainer;
+import org.apache.doris.flink.exception.DorisRuntimeException;
+import org.apache.doris.flink.tools.cdc.CdcTools;
+import org.apache.doris.flink.tools.cdc.DatabaseSyncConfig;
+import org.junit.BeforeClass;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.Semaphore;
+
+public abstract class AbstractE2EService extends AbstractContainerTestBase {
+private static final Logger LOG =
LoggerFactory.getLogger(AbstractE2EService.class);
+private static ContainerService mysqlContainerService;
+private static final CustomerSingleThreadExecutor singleThreadExecutor =
+new CustomerSingleThreadExecutor();
+protected static final Semaphore SEMAPHORE = new Semaphore(1);
+protected static final String SINK_CONF = "--" +
DatabaseSyncConfig.SINK_CONF;
+protected static final String DORIS_DATABASE = "--database";
+protected static final String HOSTNAME = "hostname";
+protected static final String PORT = "port";
+protected static final String USERNAME = "username";
+protected static final String PASSWORD = "password";
+protected static final String DATABASE_NAME = "database-name";
+protected static final String FENODES = "fenodes";
+protected static final String JDBC_URL = "jdbc-url";
+protected static final String SINK_LABEL_PREFIX = "sink.label-prefix";
+
+@BeforeClass
+public static void initE2EContainers() {
+LOG.info("Trying to Start init E2E containers.");
+initMySQLContainer();
+}
+
+private static void initMySQLContainer() {
+if (Objects.nonNull(mysqlContainerService) &&
mysqlContainerService.isRunning()) {
+LOG.info("The MySQL container has been started and is running
status.");
+return;
+}
+mysqlContainerService = new MySQLContainer();
+mysqlContainerService.startContainer();
+LOG.info("Mysql container was started.");
+}
+
+protected String getMySQLInstanceHost() {
+return mysqlContainerService.getInstanceHost();
+}
+
+protected Integer getMySQLQueryPort() {
+return mysqlContainerService.getMappedPort(3306);
+}
+
+protected String getMySQLUsername() {
+return mysqlContainerService.getUsername();
+}
+
+protected String getMySQLPassword() {
+return mysqlContainerService.getPassword();
+}
+
+protected Connection getMySQLQueryConnection() {
+return mysqlContainerService.getQueryConnection();
+}
+
+protected void submitE2EJob(String jobName, String[] args) {
+singleThreadExecutor.submitJob(
+jobName,
+() -> {
+try {
+LOG.info("{} e2e job will submit to start.", jobName);
+
CdcTools.setStreamExecutionEnvironmentForTesting(configFlinkEnvironment());
+CdcTools.main(args);
+Thread.sleep(1);
+} catch (Exception e) {
+throw new DorisRuntimeException(e);
+}
+});
+}
+
+protected void cancelCurrentE2EJob(String jobName) {
+LOG.info("{} e2e job will cancel", jobName);
+singleThreadExecutor.cancelCurrentJob(jobName);
+}
Review Comment:
[PR] [enhancement](fatal) change log fatal to throw exception to avoid core [doris]
liutang123 opened a new pull request, #40316: URL: https://github.com/apache/doris/pull/40316 back #32715 ## Proposed changes Issue Number: close #xxx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](fatal) change log fatal to throw exception to avoid core [doris]
doris-robot commented on PR #40316: URL: https://github.com/apache/doris/pull/40316#issuecomment-2325787938 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve]Transform the End-To-End(E2E) tasks on the assembly line [doris-flink-connector]
JNSimba commented on code in PR #466:
URL:
https://github.com/apache/doris-flink-connector/pull/466#discussion_r1741559069
##
flink-doris-connector/src/test/java/org/apache/doris/flink/container/AbstractE2EService.java:
##
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.container;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import org.apache.doris.flink.container.e2e.CustomerSingleThreadExecutor;
+import org.apache.doris.flink.container.instance.ContainerService;
+import org.apache.doris.flink.container.instance.MySQLContainer;
+import org.apache.doris.flink.exception.DorisRuntimeException;
+import org.apache.doris.flink.tools.cdc.CdcTools;
+import org.apache.doris.flink.tools.cdc.DatabaseSyncConfig;
+import org.junit.BeforeClass;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.Semaphore;
+
+public abstract class AbstractE2EService extends AbstractContainerTestBase {
+private static final Logger LOG =
LoggerFactory.getLogger(AbstractE2EService.class);
+private static ContainerService mysqlContainerService;
+private static final CustomerSingleThreadExecutor singleThreadExecutor =
+new CustomerSingleThreadExecutor();
+protected static final Semaphore SEMAPHORE = new Semaphore(1);
+protected static final String SINK_CONF = "--" +
DatabaseSyncConfig.SINK_CONF;
+protected static final String DORIS_DATABASE = "--database";
+protected static final String HOSTNAME = "hostname";
+protected static final String PORT = "port";
+protected static final String USERNAME = "username";
+protected static final String PASSWORD = "password";
+protected static final String DATABASE_NAME = "database-name";
+protected static final String FENODES = "fenodes";
+protected static final String JDBC_URL = "jdbc-url";
+protected static final String SINK_LABEL_PREFIX = "sink.label-prefix";
+
+@BeforeClass
+public static void initE2EContainers() {
+LOG.info("Trying to Start init E2E containers.");
+initMySQLContainer();
+}
+
+private static void initMySQLContainer() {
+if (Objects.nonNull(mysqlContainerService) &&
mysqlContainerService.isRunning()) {
+LOG.info("The MySQL container has been started and is running
status.");
+return;
+}
+mysqlContainerService = new MySQLContainer();
+mysqlContainerService.startContainer();
+LOG.info("Mysql container was started.");
+}
+
+protected String getMySQLInstanceHost() {
+return mysqlContainerService.getInstanceHost();
+}
+
+protected Integer getMySQLQueryPort() {
+return mysqlContainerService.getMappedPort(3306);
+}
+
+protected String getMySQLUsername() {
+return mysqlContainerService.getUsername();
+}
+
+protected String getMySQLPassword() {
+return mysqlContainerService.getPassword();
+}
+
+protected Connection getMySQLQueryConnection() {
+return mysqlContainerService.getQueryConnection();
+}
+
+protected void submitE2EJob(String jobName, String[] args) {
+singleThreadExecutor.submitJob(
+jobName,
+() -> {
+try {
+LOG.info("{} e2e job will submit to start.", jobName);
+
CdcTools.setStreamExecutionEnvironmentForTesting(configFlinkEnvironment());
+CdcTools.main(args);
+Thread.sleep(1);
+} catch (Exception e) {
+throw new DorisRuntimeException(e);
+}
+});
+}
+
+protected void cancelCurrentE2EJob(String jobName) {
+LOG.info("{} e2e job will cancel", jobName);
+singleThreadExecutor.cancelCurrentJob(jobName);
+}
+
+private Stre
Re: [PR] [opt](autoinc) Remove some restrictions on schema change on table that has auto-increment column [doris]
bobhan1 commented on PR #40280: URL: https://github.com/apache/doris/pull/40280#issuecomment-2325788286 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](decompressor) impl ZstdDecompressor [doris]
github-actions[bot] commented on code in PR #40315:
URL: https://github.com/apache/doris/pull/40315#discussion_r1741562099
##
be/src/exec/decompressor.h:
##
@@ -126,6 +136,25 @@ class Bzip2Decompressor : public Decompressor {
bz_stream _bz_strm;
};
+class ZstdDecompressor : public Decompressor {
+public:
+~ZstdDecompressor() override;
+
+Status decompress(uint8_t* input, size_t input_len, size_t*
input_bytes_read, uint8_t* output,
+ size_t output_max_len, size_t* decompressed_len, bool*
stream_end,
+ size_t* more_input_bytes, size_t* more_output_bytes)
override;
+
+std::string debug_info() override;
+
+private:
+friend class Decompressor;
+ZstdDecompressor() : Decompressor(CompressType::ZSTD) {}
+Status init() override;
+
+private:
Review Comment:
warning: redundant access specifier has the same accessibility as the
previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
Additional context
**be/src/exec/decompressor.h:148:** previously declared here
```cpp
private:
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhencement](sample) add delta and kudu samples [doris]
github-actions[bot] commented on PR #40067:
URL: https://github.com/apache/doris/pull/40067#issuecomment-2325790434
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/10678442605";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh line
162:
export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
^^ SC2155 (warning): Declare and assign separately to
avoid masking return values.
In samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh line
20:
export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
^^ SC2155 (warning): Declare and assign separately to
avoid masking return values.
For more information:
https://www.shellcheck.net/wiki/SC2155 -- Declare and assign separately to
...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' returned error 1 finding the following formatting issues:
--
--- samples/datalake/deltalake_and_kudu/scripts/start_doris.sh.orig
+++ samples/datalake/deltalake_and_kudu/scripts/start_doris.sh
@@ -20,8 +20,8 @@
export JAVA_HOME=/opt/jdk-17.0.2
cp -r /opt/doris-bin /opt/doris
-echo "trino_connector_plugin_dir=/opt/connectors/" >>
/opt/doris/fe/conf/fe.conf
-echo "trino_connector_plugin_dir=/opt/connectors/" >>
/opt/doris/be/conf/be.conf
+echo "trino_connector_plugin_dir=/opt/connectors/"
>>/opt/doris/fe/conf/fe.conf
+echo "trino_connector_plugin_dir=/opt/connectors/"
>>/opt/doris/be/conf/be.conf
/opt/doris/fe/bin/start_fe.sh --daemon
/opt/doris/be/bin/start_be.sh --daemon
--- samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh.orig
+++ samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh
@@ -33,7 +33,6 @@
hdfs_plugin="ff4a3e3b32dcce27f4df58f17938abde"
kudu_java_example="1afe0a890785e8d0011ea7342ae5e43d"
-
download_source_file() {
local FILE_PATH="$1"
local EXPECTED_MD5="$2"
@@ -79,9 +78,6 @@
download_source_file "trino-hdfs-435-20240724.tar.gz" "${hdfs_plugin}"
"https://github.com/apache/doris-thirdparty/releases/download/trino-435-20240724";
download_source_file "kudu-java-example-1.0-SNAPSHOT.jar"
"${kudu_java_example}"
"https://github.com/apache/doris-thirdparty/releases/download/trino-435-20240724";
-
-
-
if [[ ! -f "jdk-17.0.2/SUCCESS" ]]; then
echo "Prepare jdk17 environment"
if [[ -d "jdk-17.0.2" ]]; then
@@ -156,10 +152,9 @@
touch connectors/trino-delta-lake-435/hdfs/SUCCESS
fi
-
cd ../
-export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
+export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
docker compose -f trinoconnector-compose.yml --env-file
trinoconnector-compose.env up -d
echo "Create hive table ..."
--- samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh.orig
+++ samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh
@@ -17,6 +17,6 @@
# specific language governing permissions and limitations
# under the License.
-export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
+export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
docker compose -f trinoconnector-compose.yml --env-file
trinoconnector-compose.env down
--
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](scanner) Control the degree of parallelism of scanner when only limit involved [doris]
github-actions[bot] commented on PR #39927: URL: https://github.com/apache/doris/pull/39927#issuecomment-2325795610 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve]Transform the End-To-End(E2E) tasks on the assembly line [doris-flink-connector]
JNSimba commented on code in PR #466:
URL:
https://github.com/apache/doris-flink-connector/pull/466#discussion_r1741566913
##
flink-doris-connector/src/test/java/org/apache/doris/flink/container/e2e/Mysql2DorisE2ECase.java:
##
@@ -0,0 +1,392 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.container.e2e;
+
+import org.apache.doris.flink.container.AbstractE2EService;
+import org.apache.doris.flink.container.ContainerUtils;
+import org.apache.doris.flink.tools.cdc.DatabaseSyncConfig;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Arrays;
+import java.util.List;
+
+public class Mysql2DorisE2ECase extends AbstractE2EService {
+private static final Logger LOG =
LoggerFactory.getLogger(Mysql2DorisE2ECase.class);
+private static final String DATABASE = "test_e2e_mysql";
+private static final String CREATE_DATABASE = "CREATE DATABASE IF NOT
EXISTS " + DATABASE;
+private static final String MYSQL_CONF = "--" +
DatabaseSyncConfig.MYSQL_CONF;
+
+@Before
+public void setUp() throws InterruptedException {
+LOG.info("Mysql2DorisE2ECase attempting to acquire semaphore.");
+SEMAPHORE.acquire();
+Thread.sleep(6);
Review Comment:
What is the reason of sleep here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](scanner) Control the degree of parallelism of scanner when only limit involved [doris]
github-actions[bot] commented on PR #39927: URL: https://github.com/apache/doris/pull/39927#issuecomment-2325795676 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](autoinc) Remove some restrictions on schema change on table that has auto-increment column [doris]
github-actions[bot] commented on PR #40280: URL: https://github.com/apache/doris/pull/40280#issuecomment-2325799122 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](partial update) Remove unnecessary lock and refactor some code for partial update [doris]
bobhan1 commented on PR #40062: URL: https://github.com/apache/doris/pull/40062#issuecomment-2325799368 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhencement](sample) add delta and kudu samples [doris]
github-actions[bot] commented on PR #40067:
URL: https://github.com/apache/doris/pull/40067#issuecomment-2325807921
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/10678586462";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh line
162:
export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
^^ SC2155 (warning): Declare and assign separately to
avoid masking return values.
In samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh line
20:
export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
^^ SC2155 (warning): Declare and assign separately to
avoid masking return values.
For more information:
https://www.shellcheck.net/wiki/SC2155 -- Declare and assign separately to
...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' returned error 1 finding the following formatting issues:
--
--- samples/datalake/deltalake_and_kudu/scripts/start_doris.sh.orig
+++ samples/datalake/deltalake_and_kudu/scripts/start_doris.sh
@@ -20,8 +20,8 @@
export JAVA_HOME=/opt/jdk-17.0.2
cp -r /opt/doris-bin /opt/doris
-echo "trino_connector_plugin_dir=/opt/connectors/" >>
/opt/doris/fe/conf/fe.conf
-echo "trino_connector_plugin_dir=/opt/connectors/" >>
/opt/doris/be/conf/be.conf
+echo "trino_connector_plugin_dir=/opt/connectors/"
>>/opt/doris/fe/conf/fe.conf
+echo "trino_connector_plugin_dir=/opt/connectors/"
>>/opt/doris/be/conf/be.conf
/opt/doris/fe/bin/start_fe.sh --daemon
/opt/doris/be/bin/start_be.sh --daemon
--- samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh.orig
+++ samples/datalake/deltalake_and_kudu/start-trinoconnector-compose.sh
@@ -33,7 +33,6 @@
hdfs_plugin="ff4a3e3b32dcce27f4df58f17938abde"
kudu_java_example="1afe0a890785e8d0011ea7342ae5e43d"
-
download_source_file() {
local FILE_PATH="$1"
local EXPECTED_MD5="$2"
@@ -79,9 +78,6 @@
download_source_file "trino-hdfs-435-20240724.tar.gz" "${hdfs_plugin}"
"https://github.com/apache/doris-thirdparty/releases/download/trino-435-20240724";
download_source_file "kudu-java-example-1.0-SNAPSHOT.jar"
"${kudu_java_example}"
"https://github.com/apache/doris-thirdparty/releases/download/trino-435-20240724";
-
-
-
if [[ ! -f "jdk-17.0.2/SUCCESS" ]]; then
echo "Prepare jdk17 environment"
if [[ -d "jdk-17.0.2" ]]; then
@@ -156,10 +152,9 @@
touch connectors/trino-delta-lake-435/hdfs/SUCCESS
fi
-
cd ../
-export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
+export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
docker compose -f trinoconnector-compose.yml --env-file
trinoconnector-compose.env up -d
echo "Create hive table ..."
--- samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh.orig
+++ samples/datalake/deltalake_and_kudu/stop-trinoconnector-compose.sh
@@ -17,6 +17,6 @@
# specific language governing permissions and limitations
# under the License.
-export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
+export KUDU_QUICKSTART_IP=$(ifconfig | grep "inet " | grep -Fv 127.0.0.1 |
awk '{print $2}' | tail -1)
docker compose -f trinoconnector-compose.yml --env-file
trinoconnector-compose.env down
--
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve]Transform the End-To-End(E2E) tasks on the assembly line [doris-flink-connector]
JNSimba commented on code in PR #466:
URL:
https://github.com/apache/doris-flink-connector/pull/466#discussion_r1741576381
##
flink-doris-connector/src/test/java/org/apache/doris/flink/container/AbstractE2EService.java:
##
@@ -0,0 +1,155 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.container;
+
+import org.apache.flink.api.common.restartstrategy.RestartStrategies;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import org.apache.doris.flink.container.e2e.CustomerSingleThreadExecutor;
+import org.apache.doris.flink.container.instance.ContainerService;
+import org.apache.doris.flink.container.instance.MySQLContainer;
+import org.apache.doris.flink.exception.DorisRuntimeException;
+import org.apache.doris.flink.tools.cdc.CdcTools;
+import org.apache.doris.flink.tools.cdc.DatabaseSyncConfig;
+import org.junit.BeforeClass;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.sql.Connection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.Semaphore;
+
+public abstract class AbstractE2EService extends AbstractContainerTestBase {
+private static final Logger LOG =
LoggerFactory.getLogger(AbstractE2EService.class);
+private static ContainerService mysqlContainerService;
+private static final CustomerSingleThreadExecutor singleThreadExecutor =
+new CustomerSingleThreadExecutor();
+protected static final Semaphore SEMAPHORE = new Semaphore(1);
+protected static final String SINK_CONF = "--" +
DatabaseSyncConfig.SINK_CONF;
+protected static final String DORIS_DATABASE = "--database";
+protected static final String HOSTNAME = "hostname";
+protected static final String PORT = "port";
+protected static final String USERNAME = "username";
+protected static final String PASSWORD = "password";
+protected static final String DATABASE_NAME = "database-name";
+protected static final String FENODES = "fenodes";
+protected static final String JDBC_URL = "jdbc-url";
+protected static final String SINK_LABEL_PREFIX = "sink.label-prefix";
+
+@BeforeClass
+public static void initE2EContainers() {
+LOG.info("Trying to Start init E2E containers.");
+initMySQLContainer();
+}
+
+private static void initMySQLContainer() {
+if (Objects.nonNull(mysqlContainerService) &&
mysqlContainerService.isRunning()) {
+LOG.info("The MySQL container has been started and is running
status.");
+return;
+}
+mysqlContainerService = new MySQLContainer();
+mysqlContainerService.startContainer();
+LOG.info("Mysql container was started.");
+}
+
+protected String getMySQLInstanceHost() {
+return mysqlContainerService.getInstanceHost();
+}
+
+protected Integer getMySQLQueryPort() {
+return mysqlContainerService.getMappedPort(3306);
+}
+
+protected String getMySQLUsername() {
+return mysqlContainerService.getUsername();
+}
+
+protected String getMySQLPassword() {
+return mysqlContainerService.getPassword();
+}
+
+protected Connection getMySQLQueryConnection() {
+return mysqlContainerService.getQueryConnection();
+}
+
+protected void submitE2EJob(String jobName, String[] args) {
+singleThreadExecutor.submitJob(
+jobName,
+() -> {
+try {
+LOG.info("{} e2e job will submit to start.", jobName);
+
CdcTools.setStreamExecutionEnvironmentForTesting(configFlinkEnvironment());
+CdcTools.main(args);
+Thread.sleep(1);
Review Comment:
why sleep here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.or
Re: [PR] [opt](profile) Print information when collection of profile is not finished. [doris]
doris-robot commented on PR #40246: URL: https://github.com/apache/doris/pull/40246#issuecomment-2325810010 TPC-H: Total hot run time: 38657 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit db2407b38a3fe7067ed595aa291c95bb11d50e38, data reload: false -- Round 1 -- q1 18558 457044174417 q2 2631186 195 186 q3 11304 117411691169 q4 10596 778 732 732 q5 7813293129072907 q6 240 138 143 138 q7 994 629 605 605 q8 9315208820502050 q9 6973652765596527 q10 7040219422292194 q11 455 244 253 244 q12 394 220 223 220 q13 18388 312931143114 q14 276 236 238 236 q15 516 490 489 489 q16 577 524 523 523 q17 1003674 739 674 q18 7437683168836831 q19 1394104310181018 q20 692 331 342 331 q21 3937304430603044 q22 1109100810201008 Total cold run time: 111642 ms Total hot run time: 38657 ms - Round 2, with runtime_filter_mode=off - q1 4313430542864286 q2 374 279 269 269 q3 2949267726372637 q4 1984168716391639 q5 5416541454515414 q6 229 133 128 128 q7 2089173517361735 q8 3224337233373337 q9 8453844884318431 q10 3461324332013201 q11 582 494 514 494 q12 767 645 609 609 q13 11923 311130913091 q14 305 271 283 271 q15 534 470 474 470 q16 596 561 563 561 q17 1801150014671467 q18 7810743875857438 q19 1681153515761535 q20 2084182818081808 q21 5621508251285082 q22 1102103210091009 Total cold run time: 67298 ms Total hot run time: 54912 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ut](data type) fix get_common_type_test [doris]
eldenmoon commented on PR #40317: URL: https://github.com/apache/doris/pull/40317#issuecomment-2325810107 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [ut](data type) fix get_common_type_test [doris]
eldenmoon opened a new pull request, #40317: URL: https://github.com/apache/doris/pull/40317 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ut](data type) fix get_common_type_test [doris]
doris-robot commented on PR #40317: URL: https://github.com/apache/doris/pull/40317#issuecomment-2325810001 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](scanner) Control the degree of parallelism of scanner when only limit involved [doris]
doris-robot commented on PR #39927: URL: https://github.com/apache/doris/pull/39927#issuecomment-2325811646 TPC-H: Total hot run time: 38435 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit a9217914ff5c724d9ec5257e7280ff3896965540, data reload: false -- Round 1 -- q1 17990 452643384338 q2 2037187 196 187 q3 11294 979 1129979 q4 10181 768 770 768 q5 7753286028732860 q6 223 138 139 138 q7 965 612 602 602 q8 9330208421022084 q9 7459662765996599 q10 6995223522402235 q11 455 243 241 241 q12 407 235 226 226 q13 17775 309330643064 q14 289 253 237 237 q15 521 483 501 483 q16 584 494 500 494 q17 981 615 644 615 q18 7504692769626927 q19 13931077979 979 q20 701 337 339 337 q21 4033303830973038 q22 1124101310041004 Total cold run time: 109994 ms Total hot run time: 38435 ms - Round 2, with runtime_filter_mode=off - q1 4363434042814281 q2 381 271 283 271 q3 2857263427142634 q4 1923165917031659 q5 5630570257535702 q6 236 141 140 140 q7 2303185218551852 q8 3352346435083464 q9 8891898488958895 q10 3592340133913391 q11 616 504 513 504 q12 825 687 676 676 q13 14108 322233293222 q14 325 305 306 305 q15 538 505 502 502 q16 623 576 583 576 q17 1869156615871566 q18 8247792080447920 q19 1748160616181606 q20 2176191419741914 q21 5931556355225522 q22 1107104010901040 Total cold run time: 71641 ms Total hot run time: 57642 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](Nereids) add show statement syntax [doris]
doris-robot commented on PR #40288: URL: https://github.com/apache/doris/pull/40288#issuecomment-2325821890 TPC-H: Total hot run time: 38125 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 042a585879058c5414220ef94c18fd0b161c3587, data reload: false -- Round 1 -- q1 17603 510743154315 q2 2024181 177 177 q3 11763 964 1072964 q4 10520 682 680 680 q5 7744283428802834 q6 230 140 143 140 q7 967 618 611 611 q8 9334212021222120 q9 7317652965276527 q10 6991219321732173 q11 455 246 245 245 q12 404 225 225 225 q13 17849 309530703070 q14 278 238 233 233 q15 539 504 487 487 q16 592 515 523 515 q17 1004714 700 700 q18 7280681068626810 q19 1392102910251025 q20 688 329 345 329 q21 4317307429292929 q22 1129102510161016 Total cold run time: 110420 ms Total hot run time: 38125 ms - Round 2, with runtime_filter_mode=off - q1 4368427942934279 q2 371 273 278 273 q3 2842264326452643 q4 1901167316681668 q5 5674570058015700 q6 228 134 139 134 q7 2277186118251825 q8 3310347934803479 q9 8944878687798779 q10 3604338033523352 q11 596 510 498 498 q12 823 665 661 661 q13 15745 326832413241 q14 322 284 288 284 q15 540 490 510 490 q16 626 574 608 574 q17 1849155815771558 q18 8109790079297900 q19 1737170716341634 q20 2204189419651894 q21 5904552552875287 q22 1137103111091031 Total cold run time: 73111 ms Total hot run time: 57184 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](partial update) Fix `__DORIS_SEQUENCE_COL__` is not set for newly inserted rows in partial update [doris]
doris-robot commented on PR #40272: URL: https://github.com/apache/doris/pull/40272#issuecomment-2325821919 TPC-H: Total hot run time: 38326 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit f6d535be4be0ec683be95ba81bee79653dc63d47, data reload: false -- Round 1 -- q1 17623 456443484348 q2 2031192 175 175 q3 11733 991 1148991 q4 10513 766 678 678 q5 7773289328452845 q6 231 140 138 138 q7 981 622 596 596 q8 9339210921142109 q9 7289656866326568 q10 6988225922042204 q11 482 248 238 238 q12 395 219 219 219 q13 17767 307330813073 q14 289 226 231 226 q15 517 486 490 486 q16 595 513 510 510 q17 1007614 744 614 q18 7451685770056857 q19 1383108011041080 q20 677 338 332 332 q21 3948313630453045 q22 11121046994 994 Total cold run time: 110124 ms Total hot run time: 38326 ms - Round 2, with runtime_filter_mode=off - q1 4399435043594350 q2 382 269 265 265 q3 2851269026522652 q4 1932168416251625 q5 5705570657725706 q6 236 145 138 138 q7 2272184118211821 q8 3295344134683441 q9 8916892788398839 q10 3586338033893380 q11 621 529 503 503 q12 840 676 660 660 q13 13451 315232683152 q14 320 286 290 286 q15 536 500 476 476 q16 614 598 601 598 q17 1852155215451545 q18 8352777079837770 q19 1751174515271527 q20 2211192219171917 q21 5831548753955395 q22 1137105910541054 Total cold run time: 71090 ms Total hot run time: 57100 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ut](data type) fix get_common_type_test [doris]
github-actions[bot] commented on PR #40317: URL: https://github.com/apache/doris/pull/40317#issuecomment-2325822271 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](compaction) Support full compaction by table id for cloud [doris]
doris-robot commented on PR #39934: URL: https://github.com/apache/doris/pull/39934#issuecomment-2325822310 TPC-H: Total hot run time: 38217 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 9807c0aed409ecb5f6cb80d67d24a8ac30986249, data reload: false -- Round 1 -- q1 17642 516143164316 q2 2010187 175 175 q3 11753 963 1077963 q4 10529 687 768 687 q5 7757283828212821 q6 243 139 141 139 q7 963 627 599 599 q8 9334206620852066 q9 7184655665586556 q10 6995228422882284 q11 443 245 244 244 q12 398 217 225 217 q13 17998 308930843084 q14 281 238 228 228 q15 514 500 511 500 q16 580 519 525 519 q17 987 718 736 718 q18 7314685169526851 q19 13981155961 961 q20 680 337 343 337 q21 4006296131002961 q22 1110991 996 991 Total cold run time: 110119 ms Total hot run time: 38217 ms - Round 2, with runtime_filter_mode=off - q1 4376434243114311 q2 375 268 269 268 q3 2862267026082608 q4 1983165116931651 q5 5763571057585710 q6 231 140 143 140 q7 2225181618631816 q8 3343345034763450 q9 8813883688038803 q10 3572343433653365 q11 614 507 505 505 q12 842 660 647 647 q13 13540 327633073276 q14 337 295 305 295 q15 545 485 496 485 q16 633 603 568 568 q17 1873156415231523 q18 8166783878567838 q19 1769162815821582 q20 2218188619701886 q21 5634553353105310 q22 1116103110471031 Total cold run time: 70830 ms Total hot run time: 57068 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve]Transform the End-To-End(E2E) tasks on the assembly line [doris-flink-connector]
JNSimba commented on code in PR #466:
URL:
https://github.com/apache/doris-flink-connector/pull/466#discussion_r1741587620
##
flink-doris-connector/src/test/java/org/apache/doris/flink/container/e2e/CustomerSingleThreadExecutor.java:
##
@@ -0,0 +1,100 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.flink.container.e2e;
+
+import org.apache.doris.flink.exception.DorisRuntimeException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.concurrent.Future;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * A custom single-threaded executor service that manages the execution of
jobs in a single thread.
+ * It allows submitting a job, cancelling a currently running job, and
shutting down the executor.
+ */
+public class CustomerSingleThreadExecutor {
+private static final Logger LOG =
LoggerFactory.getLogger(CustomerSingleThreadExecutor.class);
+private final ThreadPoolExecutor executor;
+private final AtomicReference currentJobName = new
AtomicReference<>();
+private final AtomicReference> currentJob = new
AtomicReference<>();
+
+/**
+ * Constructs a new {@code CustomerSingleThreadExecutor} instance with a
single-threaded
+ * executor service.
+ */
+public CustomerSingleThreadExecutor() {
+this.executor =
+new ThreadPoolExecutor(
+1, 1, 0L, TimeUnit.MILLISECONDS, new
LinkedBlockingQueue<>());
+}
+
+/**
+ * Submits a new job for execution.
+ *
+ * @param jobName The name of the job being submitted.
+ * @param job The {@link Runnable} job to be executed.
+ * @return A {@link Future} representing the pending results of the job.
+ */
+public Future submitJob(String jobName, Runnable job) {
+try {
+LOG.info("Submitting a new job. jobName={}", jobName);
+Future future = executor.submit(job);
+currentJob.set(future);
+currentJobName.set(jobName);
+return future;
+} catch (Exception e) {
+LOG.error("Failed to submit job. jobName={}", jobName, e);
+throw new DorisRuntimeException(e);
+}
+}
+
+/**
+ * Cancels the currently running job if its name matches the provided job
name. The job is
+ * interrupted if it is currently running.
+ *
+ * @param jobName The name of the job to be cancelled.
+ */
+public void cancelCurrentJob(String jobName) {
+String currentName = currentJobName.get();
+Future currentFuture = currentJob.get();
+
+if (currentFuture != null && !currentFuture.isDone() &&
jobName.equals(currentName)) {
+LOG.info("Cancelling the current job. jobName={}", jobName);
+boolean cancelled = currentFuture.cancel(true);
+if (cancelled) {
+LOG.info("Job successfully cancelled. jobName={}", jobName);
+} else {
+LOG.info("Job cancellation failed. jobName={}", jobName);
+}
+} else {
+LOG.info("No matching job to cancel or job already completed.
jobName={}", jobName);
+}
+}
Review Comment:
Do we need to introduce a thread pool? Why not use Flink's built-in
jobClient to cancel?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](DCHECK) Fix the DCHECK failure when the query is canceled. [doris]
doris-robot commented on PR #40276: URL: https://github.com/apache/doris/pull/40276#issuecomment-2325827670 TPC-H: Total hot run time: 37989 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 97c9c4e6343572e40990a7fafae768c63fdb7d62, data reload: false -- Round 1 -- q1 17647 439242784278 q2 2027183 176 176 q3 11923 996 1105996 q4 10519 766 727 727 q5 7761282628122812 q6 230 138 138 138 q7 957 612 593 593 q8 9513205320822053 q9 7613652065546520 q10 7007225422162216 q11 466 244 254 244 q12 394 223 223 223 q13 17763 309630623062 q14 279 228 243 228 q15 514 474 476 474 q16 603 533 530 530 q17 982 685 716 685 q18 7287674667256725 q19 1395103610041004 q20 679 327 332 327 q21 4003301129762976 q22 1133104210021002 Total cold run time: 110695 ms Total hot run time: 37989 ms - Round 2, with runtime_filter_mode=off - q1 4371430743524307 q2 369 266 273 266 q3 2857264826342634 q4 1943166316451645 q5 5599571156695669 q6 245 141 136 136 q7 2229183618291829 q8 3294341134443411 q9 8882886787468746 q10 3588329333933293 q11 616 521 513 513 q12 850 611 649 611 q13 13217 334431483148 q14 310 282 291 282 q15 552 485 494 485 q16 613 590 580 580 q17 1843154614991499 q18 804777747774 q19 1756152615071507 q20 2163193919381938 q21 5773535254645352 q22 1155112511281125 Total cold run time: 70272 ms Total hot run time: 56750 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [cherry-pick](branch-2.1) Fix some group commit fault [doris]
Yukang-Lian commented on PR #40319: URL: https://github.com/apache/doris/pull/40319#issuecomment-2325829513 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv)Fix that after insert overwrite, the partition version will always be 2 [doris]
zddr commented on PR #40318: URL: https://github.com/apache/doris/pull/40318#issuecomment-2325829637 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](mtmv)Fix that after insert overwrite, the partition version will always be 2 [doris]
zddr opened a new pull request, #40318: URL: https://github.com/apache/doris/pull/40318 The replaced partition version should inherit from the original partition -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv)Fix that after insert overwrite, the partition version will always be 2 [doris]
doris-robot commented on PR #40318: URL: https://github.com/apache/doris/pull/40318#issuecomment-2325829209 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [cherry-pick](branch-2.1) Fix some group commit fault [doris]
Yukang-Lian opened a new pull request, #40319: URL: https://github.com/apache/doris/pull/40319 ## Proposed changes Pick #39986 #40120 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [cherry-pick](branch-2.1) Fix some group commit fault [doris]
doris-robot commented on PR #40319: URL: https://github.com/apache/doris/pull/40319#issuecomment-2325829397 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv) Disable sql_limit variable when query rewrite by materialize view [doris]
seawinde commented on PR #40106: URL: https://github.com/apache/doris/pull/40106#issuecomment-2325831092 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) fix bug for A>n, where A.max is infinity [doris]
doris-robot commented on PR #39936: URL: https://github.com/apache/doris/pull/39936#issuecomment-2325831387 TPC-H: Total hot run time: 38296 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 0c671a08b6b4de706f5be7dd3b5521f51f478275, data reload: false -- Round 1 -- q1 16760 436843314331 q2 2028186 181 181 q3 10461 116512191165 q4 10532 817 806 806 q5 8256292028792879 q6 236 148 146 146 q7 969 625 625 625 q8 9414211620552055 q9 7321653565086508 q10 7001220921622162 q11 462 245 251 245 q12 396 229 222 222 q13 17785 313230463046 q14 281 235 237 235 q15 539 495 491 491 q16 601 506 511 506 q17 970 675 750 675 q18 7357685668776856 q19 1385108810211021 q20 674 340 324 324 q21 3912297127982798 q22 1083103910191019 Total cold run time: 108423 ms Total hot run time: 38296 ms - Round 2, with runtime_filter_mode=off - q1 4395429042434243 q2 374 268 279 268 q3 2900264726852647 q4 1930167816991678 q5 5394541853745374 q6 222 133 133 133 q7 2099171517591715 q8 3175337233523352 q9 8417848984338433 q10 3449323731763176 q11 573 504 498 498 q12 780 622 604 604 q13 11297 311131043104 q14 313 283 295 283 q15 526 476 476 476 q16 608 569 593 569 q17 1787150914851485 q18 7738735173217321 q19 1691161214781478 q20 2063180818071807 q21 5494524353485243 q22 1129102810331028 Total cold run time: 66354 ms Total hot run time: 54915 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](profile) Print information when collection of profile is not finished. [doris]
doris-robot commented on PR #40246: URL: https://github.com/apache/doris/pull/40246#issuecomment-2325831541 TPC-DS: Total hot run time: 187549 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit db2407b38a3fe7067ed595aa291c95bb11d50e38, data reload: false query1 908 380 390 380 query2 6462204220042004 query3 6648207 215 207 query4 34035 23129 23151 23129 query5 4162491 490 490 query6 250 167 179 167 query7 4582291 285 285 query8 259 212 208 208 query9 8404244524272427 query10 422 271 264 264 query11 16417 15029 15128 15029 query12 154 100 98 98 query13 1618385 359 359 query14 10019 690975086909 query15 259 174 176 174 query16 8095466 415 415 query17 1586573 568 568 query18 2149296 296 296 query19 325 148 146 146 query20 114 110 113 110 query21 213 105 106 105 query22 4445437540814081 query23 34138 33367 33634 33367 query24 11227 296028662866 query25 657 403 405 403 query26 1159164 159 159 query27 2558282 275 275 query28 7420204320402040 query29 850 438 428 428 query30 307 163 157 157 query31 1001776 805 776 query32 99 59 60 59 query33 781 292 296 292 query34 964 484 490 484 query35 894 748 733 733 query36 963 923 923 query37 162 95 95 95 query38 4045387239053872 query39 1454137414131374 query40 207 122 119 119 query41 51 51 48 48 query42 119 94 99 94 query43 536 497 476 476 query44 1276816 745 745 query45 200 166 168 166 query46 1100739 743 739 query47 1875178418011784 query48 381 291 298 291 query49 1083467 432 432 query50 808 408 410 408 query51 6963684368906843 query52 98 86 88 86 query53 259 183 185 183 query54 1119464 454 454 query55 75 75 77 75 query56 266 251 244 244 query57 1169106910621062 query58 265 228 231 228 query59 3046299827812781 query60 292 272 263 263 query61 101 101 102 101 query62 851 637 661 637 query63 232 181 176 176 query64 5276681 653 653 query65 3226318931543154 query66 1457331 339 331 query67 15535 15223 15269 15223 query68 4484545 548 545 query69 417 275 274 274 query70 1151105311161053 query71 337 275 311 275 query72 6554406039433943 query73 749 330 338 330 query74 9132884290218842 query75 3412263326532633 query76 2337108210801080 query77 495 311 315 311 query78 9580911590509050 query79 1048529 514 514 query80 785 509 556 509 query81 570 231 237 231 query82 413 151 144 144 query83 236 159 153 153 query84 231 80 79 79 query85 1033289 279 279 query86 313 295 304 295 query87 4536438843014301 query88 2941232023452320 query89 388 282 280 280 query90 1991190 192 190 query91 125 100 100 100 query92 66 48 48 48 query93 1038533 532 532 query94 887 300 285 285 query95 352 254 254 254 query96 587 265 270 265 query97 3193302331203023 query98 221 201 197 197 query99 1486126412661264 Total cold run time: 289307 ms Total hot run time: 187549 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [fix](index compaction)Fix core when having multiple dest segments (#40254) [doris]
dataroaring merged PR #40294: URL: https://github.com/apache/doris/pull/40294 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: [fix](index compaction)Fix core when having multiple dest segments (#40254) (#40294)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 959cac45e49 [fix](index compaction)Fix core when having multiple dest
segments (#40254) (#40294)
959cac45e49 is described below
commit 959cac45e49a4b6d6982f2dcf22c0e019bc229e1
Author: qiye
AuthorDate: Tue Sep 3 15:50:35 2024 +0800
[fix](index compaction)Fix core when having multiple dest segments (#40254)
(#40294)
bp #40254
---
be/src/olap/compaction.cpp | 6 +++---
.../index_compaction/test_index_compaction_dup_keys.groovy | 2 ++
.../index_compaction/test_index_compaction_null.groovy | 3 +++
.../index_compaction/test_index_compaction_unique_keys.groovy | 2 ++
.../test_index_compaction_with_multi_index_segments.groovy | 2 ++
5 files changed, 12 insertions(+), 3 deletions(-)
diff --git a/be/src/olap/compaction.cpp b/be/src/olap/compaction.cpp
index 9109c59e8c2..d9ca73b88a1 100644
--- a/be/src/olap/compaction.cpp
+++ b/be/src/olap/compaction.cpp
@@ -686,9 +686,9 @@ Status Compaction::do_inverted_index_compaction() {
<< st;
return st;
}
-for (const auto& writer : inverted_index_file_writers) {
-writer->set_file_writer_opts(ctx.get_file_writer_options());
-}
+}
+for (const auto& writer : inverted_index_file_writers) {
+writer->set_file_writer_opts(ctx.get_file_writer_options());
}
// use tmp file dir to store index files
diff --git
a/regression-test/suites/inverted_index_p0/index_compaction/test_index_compaction_dup_keys.groovy
b/regression-test/suites/inverted_index_p0/index_compaction/test_index_compaction_dup_keys.groovy
index 3e1cbc6b6e2..b181ede9eb6 100644
---
a/regression-test/suites/inverted_index_p0/index_compaction/test_index_compaction_dup_keys.groovy
+++
b/regression-test/suites/inverted_index_p0/index_compaction/test_index_compaction_dup_keys.groovy
@@ -24,6 +24,7 @@ suite("test_index_compaction_dup_keys", "nonConcurrent") {
def backendId_to_backendHttpPort = [:]
getBackendIpHttpPort(backendId_to_backendIP, backendId_to_backendHttpPort);
+sql """ set global enable_match_without_inverted_index = false """
boolean disableAutoCompaction = false
def set_be_config = { key, value ->
@@ -239,5 +240,6 @@ suite("test_index_compaction_dup_keys", "nonConcurrent") {
if (has_update_be_config) {
set_be_config.call("inverted_index_compaction_enable",
invertedIndexCompactionEnable.toString())
}
+sql """ set global enable_match_without_inverted_index = true """
}
}
diff --git
a/regression-test/suites/inverted_index_p0/index_compaction/test_index_compaction_null.groovy
b/regression-test/suites/inverted_index_p0/index_compaction/test_index_compaction_null.groovy
index 2a0647ecb63..e9b4294c4df 100644
---
a/regression-test/suites/inverted_index_p0/index_compaction/test_index_compaction_null.groovy
+++
b/regression-test/suites/inverted_index_p0/index_compaction/test_index_compaction_null.groovy
@@ -24,6 +24,7 @@ suite("test_index_compaction_null", "nonConcurrent") {
def backendId_to_backendHttpPort = [:]
getBackendIpHttpPort(backendId_to_backendIP, backendId_to_backendHttpPort);
+sql """ set global enable_match_without_inverted_index = false """
boolean disableAutoCompaction = false
def set_be_config = { key, value ->
@@ -316,5 +317,7 @@ suite("test_index_compaction_null", "nonConcurrent") {
if (has_update_be_config) {
set_be_config.call("inverted_index_compaction_enable",
invertedIndexCompactionEnable.toString())
}
+
+sql """ set global enable_match_without_inverted_index = true """
}
}
diff --git
a/regression-test/suites/inverted_index_p0/index_compaction/test_index_compaction_unique_keys.groovy
b/regression-test/suites/inverted_index_p0/index_compaction/test_index_compaction_unique_keys.groovy
index 9d5f3b4cef6..1854be6d674 100644
---
a/regression-test/suites/inverted_index_p0/index_compaction/test_index_compaction_unique_keys.groovy
+++
b/regression-test/suites/inverted_index_p0/index_compaction/test_index_compaction_unique_keys.groovy
@@ -24,6 +24,7 @@ suite("test_index_compaction_unique_keys", "nonConcurrent") {
def backendId_to_backendHttpPort = [:]
getBackendIpHttpPort(backendId_to_backendIP, backendId_to_backendHttpPort);
+sql """ set global enable_match_without_inverted_index = false """
boolean disableAutoCompaction = false
def set_be_config = { key, value ->
@@ -245,5 +246,6 @@ suite("test_index_compaction_unique_keys", "nonConcurrent")
{
if (has_update_be_config) {
set_be_config.call("inverted_index_compaction_enable",
invertedIndex
Re: [PR] [opt](function)Optimize the performance of the pad function under UTF-8. [doris]
HappenLee commented on code in PR #40162:
URL: https://github.com/apache/doris/pull/40162#discussion_r1741594311
##
be/src/vec/functions/function_string.h:
##
@@ -1556,86 +1556,110 @@ class FunctionStringPad : public IFunction {
const auto* padcol = assert_cast(col[2].get());
const auto& padcol_offsets = padcol->get_offsets();
const auto& padcol_chars = padcol->get_chars();
+std::visit(
+[&](auto str_const, auto len_const, auto pad_const) {
+execute_utf8(
+strcol_offsets, strcol_chars, col_len_data,
padcol_offsets,
+padcol_chars, res_offsets, res_chars,
null_map_data, input_rows_count);
+},
+vectorized::make_bool_variant(col_const[0]),
+vectorized::make_bool_variant(col_const[1]),
+vectorized::make_bool_variant(col_const[2]));
-std::vector str_index;
+block.get_by_position(result).column =
+ColumnNullable::create(std::move(res), std::move(null_map));
+return Status::OK();
+}
+
+template
+void execute_utf8(const ColumnString::Offsets& strcol_offsets,
+ const ColumnString::Chars& strcol_chars,
+ const ColumnInt32::Container& col_len_data,
+ const ColumnString::Offsets& padcol_offsets,
+ const ColumnString::Chars& padcol_chars,
ColumnString::Offsets& res_offsets,
+ ColumnString::Chars& res_chars, ColumnUInt8::Container&
null_map_data,
+ size_t input_rows_count) const {
std::vector pad_index;
+size_t const_pad_char_size = 0;
+// If pad_const = true, initialize pad_index only once.
+// The same logic applies to the if constexpr (!pad_const) condition
below.
+if constexpr (pad_const) {
+const_pad_char_size = simd::VStringFunctions::get_char_len(
+(const char*)padcol_chars.data(), padcol_offsets[0],
pad_index);
+}
fmt::memory_buffer buffer;
-const bool str_const = col_const[0];
-const bool len_const = col_const[1];
-const bool pad_const = col_const[2];
+buffer.reserve(strcol_chars.size());
+size_t buffer_len = 0;
+
for (size_t i = 0; i < input_rows_count; ++i) {
-str_index.clear();
-pad_index.clear();
+if constexpr (!pad_const) {
+pad_index.clear();
+}
buffer.clear();
-const auto len = col_len_data[index_check_const(i, len_const)];
+const auto len = col_len_data[index_check_const(i)];
if (len < 0) {
// return NULL when input length is invalid number
null_map_data[i] = true;
-StringOP::push_empty_string(i, res_chars, res_offsets);
+res_offsets[i] = buffer_len;
} else {
-const auto str_idx = index_check_const(i, str_const);
+const auto str_idx = index_check_const(i);
const int str_len = strcol_offsets[str_idx] -
strcol_offsets[str_idx - 1];
const auto* str_data = &strcol_chars[strcol_offsets[str_idx -
1]];
-const auto pad_idx = index_check_const(i, pad_const);
+const auto pad_idx = index_check_const(i);
const int pad_len = padcol_offsets[pad_idx] -
padcol_offsets[pad_idx - 1];
const auto* pad_data = &padcol_chars[padcol_offsets[pad_idx -
1]];
-// get utf8 len
-size_t str_char_size =
simd::VStringFunctions::get_char_len((const char*)str_data,
-
str_len, str_index);
-size_t pad_char_size =
simd::VStringFunctions::get_char_len((const char*)pad_data,
-
pad_len, pad_index);
-
-if (len <= str_char_size) {
-// truncate the input string
-if (len < str_char_size) {
-buffer.append(str_data, str_data + str_index[len]);
-} else {
-buffer.append(str_data, str_data + str_len);
-}
-
StringOP::push_value_string(std::string_view(buffer.data(), buffer.size()), i,
-res_chars, res_offsets);
+auto [iterate_byte_len, iterate_char_len] =
+simd::VStringFunctions::iterate_utf8_with_limit_length(
+(const char*)str_data, (const char*)str_data +
str_len, len);
+// If iterate_char_len is greater than len, it indicates that
the string's length exceeds len, so truncation.
+if (itera
Re: [PR] [opt](scanner) Control the degree of parallelism of scanner when only limit involved [doris]
doris-robot commented on PR #39927: URL: https://github.com/apache/doris/pull/39927#issuecomment-2325834128 TPC-DS: Total hot run time: 192721 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit a9217914ff5c724d9ec5257e7280ff3896965540, data reload: false query1 1286899 862 862 query2 6386193418901890 query3 10636 388939343889 query4 59568 27317 23236 23236 query5 5457503 507 503 query6 396 161 178 161 query7 5816293 297 293 query8 291 223 213 213 query9 9075242424252424 query10 477 276 258 258 query11 18050 15035 15515 15035 query12 161 100 101 100 query13 1569413 401 401 query14 10267 774074737473 query15 258 175 174 174 query16 7082526 494 494 query17 1187590 606 590 query18 1636318 328 318 query19 307 150 150 150 query20 127 116 120 116 query21 215 111 107 107 query22 4650440648864406 query23 34298 33792 33594 33594 query24 5976288428592859 query25 557 403 394 394 query26 693 156 155 155 query27 1789280 277 277 query28 3768205120332033 query29 678 425 429 425 query30 233 154 152 152 query31 948 753 753 753 query32 89 57 53 53 query33 462 297 303 297 query34 876 475 475 475 query35 825 697 745 697 query36 1087909 959 909 query37 150 95 90 90 query38 4061392939313929 query39 1487139613781378 query40 197 119 118 118 query41 49 47 46 46 query42 120 98 96 96 query43 497 467 469 467 query44 1088732 732 732 query45 192 169 165 165 query46 1102766 750 750 query47 1906177318171773 query48 389 293 290 290 query49 777 437 447 437 query50 834 421 421 421 query51 7038696768346834 query52 97 87 87 87 query53 253 180 176 176 query54 569 475 467 467 query55 78 77 73 73 query56 283 265 273 265 query57 1197107810821078 query58 224 241 239 239 query59 3027281528722815 query60 293 267 278 267 query61 101 97 100 97 query62 734 666 629 629 query63 211 180 181 180 query64 2880687 668 668 query65 3234313632033136 query66 691 345 370 345 query67 15561 15475 15147 15147 query68 4466559 566 559 query69 430 273 276 273 query70 1078101011101010 query71 368 272 284 272 query72 6479406239883988 query73 751 329 329 329 query74 9163877788568777 query75 3369267426482648 query76 1829109210291029 query77 565 308 318 308 query78 9776906889678967 query79 1071530 515 515 query80 915 487 506 487 query81 558 241 232 232 query82 241 143 143 143 query83 228 148 151 148 query84 263 73 75 73 query85 900 280 326 280 query86 348 281 301 281 query87 4443425943284259 query88 3013233423292329 query89 384 285 282 282 query90 1892195 187 187 query91 122 97 98 97 query92 59 47 48 47 query93 1855527 535 527 query94 743 282 303 282 query95 351 256 255 255 query96 599 265 263 263 query97 3221305630693056 query98 221 204 198 198 query99 1567127012591259 Total cold run time: 307764 ms Total hot run time: 192721 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [feature](nereids)support correlated scalar subquery without scalar agg [doris]
starocean999 commented on PR #39471: URL: https://github.com/apache/doris/pull/39471#issuecomment-2325835120 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](regression-test) fix error url check in test_etl_failed case for cloud p0 [doris]
github-actions[bot] commented on PR #40298: URL: https://github.com/apache/doris/pull/40298#issuecomment-2325836109 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](regression-test) fix error url check in test_etl_failed case for cloud p0 [doris]
github-actions[bot] commented on PR #40298: URL: https://github.com/apache/doris/pull/40298#issuecomment-2325836188 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](testcase) fix unstable case [doris]
github-actions[bot] commented on PR #40124: URL: https://github.com/apache/doris/pull/40124#issuecomment-2325836352 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](testcase) fix unstable case [doris]
github-actions[bot] commented on PR #40124: URL: https://github.com/apache/doris/pull/40124#issuecomment-2325836415 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: [test](inverted index)Add upgrade and downgrade test cases for inverted_index (#37783)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new d8aeb3cbe7e [test](inverted index)Add upgrade and downgrade test cases
for inverted_index (#37783)
d8aeb3cbe7e is described below
commit d8aeb3cbe7e852e4af0a92c464df14eed894a7aa
Author: ChenPeng2013
AuthorDate: Fri Aug 30 10:19:34 2024 +0800
[test](inverted index)Add upgrade and downgrade test cases for
inverted_index (#37783)
## Proposed changes
Issue Number: close #xxx
add case
---
.../inverted_index.groovy | 213 +
.../suites/inverted_index_up_down_p0/load.groovy | 113 +++
2 files changed, 326 insertions(+)
diff --git
a/regression-test/suites/inverted_index_up_down_p0/inverted_index.groovy
b/regression-test/suites/inverted_index_up_down_p0/inverted_index.groovy
new file mode 100644
index 000..913602905e7
--- /dev/null
+++ b/regression-test/suites/inverted_index_up_down_p0/inverted_index.groovy
@@ -0,0 +1,213 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_upgrade_downgrade_compatibility_inverted_index","p0,inverted_index,restart_fe")
{
+def timeout = 12
+def delta_time = 1000
+def alter_res = "null"
+def useTime = 0
+
+sql "SET enable_match_without_inverted_index = false"
+
+def wait_for_latest_op_on_table_finish = { table_name, OpTimeout ->
+for(int t = delta_time; t <= OpTimeout; t += delta_time){
+alter_res = sql """SHOW ALTER TABLE COLUMN WHERE TableName =
"${table_name}" ORDER BY CreateTime DESC LIMIT 1;"""
+alter_res = alter_res.toString()
+if(alter_res.contains("FINISHED")) {
+sleep(3000) // wait change table state to normal
+logger.info(table_name + " latest alter job finished, detail:
" + alter_res)
+break
+}
+useTime = t
+sleep(delta_time)
+}
+assertTrue(useTime <= OpTimeout, "wait_for_latest_op_on_table_finish
timeout")
+}
+
+def wait_for_build_index_on_partition_finish = { table_name, OpTimeout ->
+for(int t = delta_time; t <= OpTimeout; t += delta_time){
+alter_res = sql """SHOW BUILD INDEX WHERE TableName =
"${table_name}";"""
+def expected_finished_num = alter_res.size();
+def finished_num = 0;
+for (int i = 0; i < expected_finished_num; i++) {
+logger.info(table_name + " build index job state: " +
alter_res[i][7] + i)
+if (alter_res[i][7] == "FINISHED") {
+++finished_num;
+}
+}
+if (finished_num == expected_finished_num) {
+logger.info(table_name + " all build index jobs finished,
detail: " + alter_res)
+break
+}
+useTime = t
+sleep(delta_time)
+}
+assertTrue(useTime <= OpTimeout,
"wait_for_latest_build_index_on_partition_finish timeout")
+}
+
+for (version in ["V1", "V2"]) {
+// duplicate table
+def tableName = "t_up_down_inverted_index${version}_duplicate"
+def result = sql(String.format("show create table %s", tableName))
+if
(!result[0][1].toString().contains(String.format('''"inverted_index_storage_format"
= "%s"''', version))) {
+throw new IllegalStateException("${tableName}
inverted_index_storage_format exception")
+}
+result = sql("select count(*) from ${tableName}")
+if (result[0][0] != 10) {
+throw new IllegalStateException("count error")
+}
+result = sql("select count(*) from ${tableName} where en in ('I see',
'Let go')")
+if (result[0][0] != 2) {
+throw new IllegalStateException("count error")
+}
+result = sql("select count(*) from ${tableName} where en match_any
('on')")
+if (result[0][0] != 2) {
+throw new IllegalStateException("count error")
+}
+
Re: [PR] [fix](nereids) fix bug for A>n, where A.max is infinity [doris]
github-actions[bot] commented on PR #39936: URL: https://github.com/apache/doris/pull/39936#issuecomment-2325840674 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) fix bug for A>n, where A.max is infinity [doris]
github-actions[bot] commented on PR #39936: URL: https://github.com/apache/doris/pull/39936#issuecomment-2325840757 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: [opt](routine load) support routine load perceived schema change (#39412)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 0b52b7ee7a2 [opt](routine load) support routine load perceived schema
change (#39412)
0b52b7ee7a2 is described below
commit 0b52b7ee7a219d7750295ef7b17c32cdc7ee26a8
Author: hui lai <1353307...@qq.com>
AuthorDate: Tue Aug 27 20:29:44 2024 +0800
[opt](routine load) support routine load perceived schema change (#39412)
At present, if the table structure changes, the routine load cannot
perceive it. As a long-running load, it should be able to perceive the
changes in the table structure.
---
.../load/routineload/KafkaRoutineLoadJob.java | 1 -
.../doris/load/routineload/KafkaTaskInfo.java | 8 +-
.../doris/load/routineload/RoutineLoadJob.java | 19 +--
.../routine_load/test_routine_load_with_sc.out | 4 +
.../data/test_routine_load_with_sc.csv | 1 +
.../routine_load/test_routine_load_with_sc.groovy | 149 +
6 files changed, 163 insertions(+), 19 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java
index abd1800a19d..1762f8d1122 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java
@@ -174,7 +174,6 @@ public class KafkaRoutineLoadJob extends RoutineLoadJob {
@Override
public void prepare() throws UserException {
-super.prepare();
// should reset converted properties each time the job being prepared.
// because the file info can be changed anytime.
convertCustomProperties(true);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaTaskInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaTaskInfo.java
index dedb66d5f94..52a1ad8559f 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaTaskInfo.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaTaskInfo.java
@@ -19,10 +19,12 @@ package org.apache.doris.load.routineload;
import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.Table;
import org.apache.doris.common.Config;
import org.apache.doris.common.UserException;
import org.apache.doris.common.util.DebugUtil;
+import org.apache.doris.planner.StreamLoadPlanner;
import org.apache.doris.thrift.TFileFormatType;
import org.apache.doris.thrift.TKafkaLoadInfo;
import org.apache.doris.thrift.TLoadSourceType;
@@ -127,7 +129,11 @@ public class KafkaTaskInfo extends RoutineLoadTaskInfo {
private TPipelineFragmentParams rePlan(RoutineLoadJob routineLoadJob)
throws UserException {
TUniqueId loadId = new TUniqueId(id.getMostSignificantBits(),
id.getLeastSignificantBits());
// plan for each task, in case table has change(rollup or schema
change)
-TPipelineFragmentParams tExecPlanFragmentParams =
routineLoadJob.plan(loadId, txnId);
+Database db =
Env.getCurrentInternalCatalog().getDbOrMetaException(routineLoadJob.getDbId());
+StreamLoadPlanner planner = new StreamLoadPlanner(db,
+(OlapTable)
db.getTableOrMetaException(routineLoadJob.getTableId(),
+Table.TableType.OLAP), routineLoadJob);
+TPipelineFragmentParams tExecPlanFragmentParams =
routineLoadJob.plan(planner, loadId, txnId);
TPlanFragment tPlanFragment = tExecPlanFragmentParams.getFragment();
tPlanFragment.getOutputSink().getOlapTableSink().setTxnId(txnId);
// it needs update timeout to make task timeout backoff work
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
index 2b8cbbd81ac..b983d6beed4 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadJob.java
@@ -262,9 +262,6 @@ public abstract class RoutineLoadJob
// The tasks belong to this job
protected List routineLoadTaskInfoList =
Lists.newArrayList();
-// stream load planer will be initialized during job schedule
-protected StreamLoadPlanner planner;
-
// this is the origin stmt of CreateRoutineLoadStmt, we use it to persist
the RoutineLoadJob,
// because we can not serialize the Expressions contained in job.
@SerializedName("ostmt")
@@ -967,21 +964,9 @@ public abstract class RoutineLoadJob
// call before first scheduling
// derived class can
Re: [PR] [opt](profile) Print information when collection of profile is not finished. [doris]
doris-robot commented on PR #40246: URL: https://github.com/apache/doris/pull/40246#issuecomment-2325841667 ClickBench: Total hot run time: 32.42 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit db2407b38a3fe7067ed595aa291c95bb11d50e38, data reload: false query1 0.050.040.04 query2 0.090.040.05 query3 0.230.050.05 query4 1.680.070.07 query5 0.480.490.49 query6 1.130.730.71 query7 0.020.020.01 query8 0.050.040.04 query9 0.550.490.48 query10 0.550.540.55 query11 0.160.120.12 query12 0.150.120.12 query13 0.600.590.58 query14 2.052.052.10 query15 0.900.820.81 query16 0.360.380.38 query17 1.071.061.02 query18 0.220.210.20 query19 1.821.791.73 query20 0.010.010.01 query21 15.40 0.650.65 query22 4.107.162.34 query23 18.31 1.481.29 query24 2.070.260.23 query25 0.160.090.08 query26 0.260.180.18 query27 0.080.080.08 query28 13.13 1.051.02 query29 12.61 3.373.32 query30 0.240.060.05 query31 2.900.410.40 query32 3.230.500.48 query33 2.913.112.99 query34 17.03 4.384.39 query35 4.444.424.39 query36 0.660.460.47 query37 0.180.150.16 query38 0.150.140.15 query39 0.050.040.05 query40 0.160.130.13 query41 0.100.050.05 query42 0.050.040.05 query43 0.050.040.04 Total cold run time: 110.44 s Total hot run time: 32.42 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: [chore](conf) Specify UTF8 as the default charset. (#39521)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 2899c78a6ac [chore](conf) Specify UTF8 as the default charset. (#39521)
2899c78a6ac is described below
commit 2899c78a6ac234bfecf27c99182a57065f59bf98
Author: yagagagaga
AuthorDate: Mon Sep 2 13:11:58 2024 +0800
[chore](conf) Specify UTF8 as the default charset. (#39521)
According the [JEP 400](https://openjdk.org/jeps/400), UTF-8 was be used
as the default charset of the JavaSE 18. But when you use the version
below 18, the default charset depends on your locale. Usually, it can
work well in many sence, but Apache Doris only supports UTF-8 as its
charset, which may occur some decoding incorrectly.
So it is necessary to set UTF8 as the default JDK charset.
---
conf/be.conf | 4 ++--
conf/fe.conf | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/conf/be.conf b/conf/be.conf
index 09b4d59daee..c69dd99699d 100644
--- a/conf/be.conf
+++ b/conf/be.conf
@@ -21,10 +21,10 @@ CUR_DATE=`date +%Y%m%d-%H%M%S`
LOG_DIR="${DORIS_HOME}/log/"
# For jdk 8
-JAVA_OPTS="-Xmx1024m -DlogPath=$LOG_DIR/jni.log
-Xloggc:$LOG_DIR/be.gc.log.$CUR_DATE -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=50M
-Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true
-Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
+JAVA_OPTS="-Dfile.encoding=UTF-8 -Xmx1024m -DlogPath=$LOG_DIR/jni.log
-Xloggc:$LOG_DIR/be.gc.log.$CUR_DATE -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=50M
-Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true
-Dsun.java.command=DorisBE -XX:-CriticalJNINatives"
# For jdk 17, this JAVA_OPTS will be used as default JVM options
-JAVA_OPTS_FOR_JDK_17="-Xmx1024m -DlogPath=$LOG_DIR/jni.log
-Xlog:gc*:$LOG_DIR/be.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
-Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true
-Dsun.java.command=DorisBE -XX:-CriticalJNINatives
-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED --ad [...]
+JAVA_OPTS_FOR_JDK_17="-Dfile.encoding=UTF-8 -Xmx1024m
-DlogPath=$LOG_DIR/jni.log
-Xlog:gc*:$LOG_DIR/be.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
-Djavax.security.auth.useSubjectCredsOnly=false -Dsun.security.krb5.debug=true
-Dsun.java.command=DorisBE -XX:-CriticalJNINatives
-XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.lang.invoke=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/ja
[...]
# Set your own JAVA_HOME
# JAVA_HOME=/path/to/jdk/
diff --git a/conf/fe.conf b/conf/fe.conf
index 815f96f9d34..bbfad0c15a8 100644
--- a/conf/fe.conf
+++ b/conf/fe.conf
@@ -27,10 +27,10 @@ CUR_DATE=`date +%Y%m%d-%H%M%S`
LOG_DIR = ${DORIS_HOME}/log
# For jdk 8
-JAVA_OPTS="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m
-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:MaxGCPauseMillis=200
-XX:+PrintGCDateStamps -XX:+PrintGCDetails
-Xloggc:$LOG_DIR/log/fe.gc.log.$CUR_DATE -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=50M
-Dlog4j2.formatMsgNoLookups=true"
+JAVA_OPTS="-Dfile.encoding=UTF-8
-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m
-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:MaxGCPauseMillis=200
-XX:+PrintGCDateStamps -XX:+PrintGCDetails
-Xloggc:$LOG_DIR/log/fe.gc.log.$CUR_DATE -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=50M
-Dlog4j2.formatMsgNoLookups=true"
# For jdk 17, this JAVA_OPTS will be used as default JVM options
-JAVA_OPTS_FOR_JDK_17="-Djavax.security.auth.useSubjectCredsOnly=false
-Xmx8192m -Xms8192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR
-Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED"
+JAVA_OPTS_FOR_JDK_17="-Dfile.encoding=UTF-8
-Djavax.security.auth.useSubjectCredsOnly=false -Xmx8192m -Xms8192m
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$LOG_DIR
-Xlog:gc*:$LOG_DIR/fe.gc.log.$CUR_DATE:time,uptime:filecount=10,filesize=50M
--add-opens=java.base/java.nio=ALL-UNNAMED --add-opens
java.base/jdk.internal.ref=ALL-UNNAMED"
# Set your own JAVA_HOME
# JAVA_HOME=/path/to/jdk/
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feat](nereids) add max/min filter push down rewrite rule [doris]
morrySnow commented on PR #39252: URL: https://github.com/apache/doris/pull/39252#issuecomment-2325844019 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](scanner) Control the degree of parallelism of scanner when only limit involved [doris]
doris-robot commented on PR #39927: URL: https://github.com/apache/doris/pull/39927#issuecomment-2325844263 ClickBench: Total hot run time: 32.77 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit a9217914ff5c724d9ec5257e7280ff3896965540, data reload: false query1 0.040.050.04 query2 0.070.040.04 query3 0.220.050.05 query4 1.670.080.08 query5 0.490.490.50 query6 1.130.730.74 query7 0.020.020.02 query8 0.050.040.04 query9 0.560.490.50 query10 0.550.530.52 query11 0.150.110.11 query12 0.140.130.12 query13 0.610.590.58 query14 2.042.072.11 query15 0.880.830.83 query16 0.370.380.37 query17 0.991.041.01 query18 0.220.210.21 query19 1.831.731.79 query20 0.020.010.02 query21 15.40 0.660.65 query22 4.076.692.57 query23 18.20 1.331.29 query24 2.150.230.22 query25 0.160.090.09 query26 0.270.170.18 query27 0.080.080.08 query28 13.19 1.010.99 query29 12.61 3.373.36 query30 0.240.060.05 query31 2.880.400.39 query32 3.260.490.47 query33 2.943.023.04 query34 17.12 4.384.40 query35 4.434.434.42 query36 0.670.480.48 query37 0.190.150.15 query38 0.150.140.15 query39 0.050.040.04 query40 0.160.130.13 query41 0.100.050.05 query42 0.060.060.05 query43 0.050.040.04 Total cold run time: 110.48 s Total hot run time: 32.77 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](Nereids) add show statement syntax [doris]
doris-robot commented on PR #40288: URL: https://github.com/apache/doris/pull/40288#issuecomment-232588 TPC-DS: Total hot run time: 192317 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 042a585879058c5414220ef94c18fd0b161c3587, data reload: false query1 1250869 869 869 query2 6389206620192019 query3 10630 397937223722 query4 59474 26412 23171 23171 query5 5431495 498 495 query6 404 157 153 153 query7 5770303 293 293 query8 292 214 207 207 query9 8845244824432443 query10 487 268 263 263 query11 17665 15005 15388 15005 query12 171 103 103 103 query13 1528401 396 396 query14 11194 726173847261 query15 250 187 190 187 query16 7506465 412 412 query17 1100589 620 589 query18 1910295 307 295 query19 300 156 151 151 query20 130 112 120 112 query21 213 104 105 104 query22 4648471344174417 query23 34092 33549 33412 33412 query24 5946283728202820 query25 546 405 400 400 query26 677 156 154 154 query27 1778283 281 281 query28 3603205320162016 query29 678 429 432 429 query30 235 151 150 150 query31 916 762 793 762 query32 78 63 52 52 query33 479 295 301 295 query34 866 492 469 469 query35 854 710 738 710 query36 1039970 949 949 query37 150 93 97 93 query38 3964390638613861 query39 1437141113831383 query40 205 121 115 115 query41 48 48 47 47 query42 113 94 93 93 query43 519 482 483 482 query44 1103732 742 732 query45 198 169 165 165 query46 1090733 749 733 query47 1904180118641801 query48 377 293 299 293 query49 779 435 431 431 query50 817 415 422 415 query51 7028693667446744 query52 100 89 90 89 query53 260 180 181 180 query54 573 472 458 458 query55 81 78 79 78 query56 290 269 272 269 query57 1215107510691069 query58 226 245 225 225 query59 3066298829132913 query60 297 266 289 266 query61 104 104 96 96 query62 782 647 642 642 query63 218 184 192 184 query64 2740668 665 665 query65 3214314131903141 query66 684 348 335 335 query67 15331 15386 15087 15087 query68 4415564 547 547 query69 429 284 285 284 query70 1186112911441129 query71 369 272 275 272 query72 6584400740214007 query73 761 329 324 324 query74 9228887989468879 query75 3354265227652652 query76 1935105410281028 query77 561 318 312 312 query78 10098 907390179017 query79 1071545 530 530 query80 834 498 509 498 query81 565 232 233 232 query82 330 140 142 140 query83 255 148 148 148 query84 260 75 79 75 query85 910 284 278 278 query86 343 300 296 296 query87 4411420243594202 query88 3069232323352323 query89 385 286 281 281 query90 2046195 190 190 query91 125 101 102 101 query92 60 51 51 51 query93 1240517 522 517 query94 766 309 294 294 query95 341 252 259 252 query96 588 267 262 262 query97 3242306330823063 query98 223 209 200 200 query99 1556128212891282 Total cold run time: 307827 ms Total hot run time: 192317 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [Fix](partial update) Fix `__DORIS_SEQUENCE_COL__` is not set for newly inserted rows in partial update [doris]
doris-robot commented on PR #40272: URL: https://github.com/apache/doris/pull/40272#issuecomment-2325844493 TPC-DS: Total hot run time: 192716 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit f6d535be4be0ec683be95ba81bee79653dc63d47, data reload: false query1 1265900 872 872 query2 6275200020262000 query3 10750 432440224022 query4 59819 26879 23309 23309 query5 5416495 514 495 query6 421 160 155 155 query7 5764296 292 292 query8 295 214 216 214 query9 9019243624502436 query10 505 276 260 260 query11 18369 15071 15601 15071 query12 164 114 109 109 query13 1589391 380 380 query14 11254 759570467046 query15 252 179 181 179 query16 7472498 493 493 query17 1128605 588 588 query18 1923304 297 297 query19 300 152 147 147 query20 117 114 110 110 query21 212 104 103 103 query22 4717455445624554 query23 34424 33378 33523 33378 query24 5929284727952795 query25 546 376 384 376 query26 683 150 150 150 query27 1805274 281 274 query28 3800203020122012 query29 672 396 394 394 query30 238 150 152 150 query31 933 762 754 754 query32 80 49 52 49 query33 441 278 276 276 query34 854 472 473 472 query35 821 717 728 717 query36 1042956 939 939 query37 145 92 84 84 query38 3979388639103886 query39 1440139214031392 query40 196 112 112 112 query41 46 50 48 48 query42 110 94 94 94 query43 511 499 467 467 query44 1053717 719 717 query45 197 164 163 163 query46 1095731 746 731 query47 1856176518131765 query48 379 299 296 296 query49 747 419 442 419 query50 829 422 418 418 query51 7102699569596959 query52 102 89 88 88 query53 248 182 180 180 query54 567 469 464 464 query55 74 73 72 72 query56 284 280 270 270 query57 1187109910551055 query58 225 232 254 232 query59 3111287830062878 query60 299 283 279 279 query61 125 121 129 121 query62 744 658 632 632 query63 219 182 192 182 query64 2936686 654 654 query65 3197316531493149 query66 696 331 342 331 query67 15326 15289 15271 15271 query68 3017557 562 557 query69 405 277 275 275 query70 1195103311071033 query71 358 272 270 270 query72 6195397841533978 query73 755 324 330 324 query74 9278884988678849 query75 3389263926922639 query76 1408981 983 981 query77 540 316 306 306 query78 9819908190859081 query79 1047534 520 520 query80 692 494 496 494 query81 467 237 234 234 query82 241 141 140 140 query83 174 149 145 145 query84 252 76 74 74 query85 675 289 283 283 query86 325 293 286 286 query87 4532431442114211 query88 3398230723672307 query89 383 278 281 278 query90 1911187 184 184 query91 122 97 101 97 query92 56 48 47 47 query93 1046529 532 529 query94 694 274 298 274 query95 350 251 249 249 query96 590 262 264 262 query97 3180309830663066 query98 217 201 204 201 query99 1501128012651265 Total cold run time: 306531 ms Total hot run time: 192716 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
(doris) branch branch-3.0 updated (2899c78a6ac -> f925b6ea705)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
from 2899c78a6ac [chore](conf) Specify UTF8 as the default charset. (#39521)
new bef810ad6fe [Fix](parquet-reader) Fix definition level rle decode dead
loop in parquet-reader. (#39523)
new f925b6ea705 [Enhancement](txn) Block new insert into if schema change
happens during transaction (#39483)
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/olap/schema_change.cpp | 1 +
be/src/util/bit_stream_utils.h | 4 +
be/src/util/rle_encoding.h | 2 +
be/src/vec/exec/format/parquet/level_decoder.h | 4 +-
.../exec/format/parquet/vparquet_column_reader.cpp | 20 +++-
.../insert/BatchInsertIntoTableCommand.java| 11 ++
.../trees/plans/physical/PhysicalResultSink.java | 4 -
.../java/org/apache/doris/qe/StmtExecutor.java | 1 +
.../apache/doris/transaction/TransactionEntry.java | 18 +++
.../data/insert_p0/{ => transaction}/test_txn.out | 0
.../insert_p0/{ => transaction}/txn_insert.out | 0
.../{ => transaction}/txn_insert_inject_case.out | 0
.../{ => transaction}/txn_insert_restart_fe.out| 0
.../txn_insert_restart_fe_with_schema_change.out | 0
.../txn_insert_values_with_schema_change.out | 0
.../{ => transaction}/txn_insert_with_drop.out | 0
.../txn_insert_with_schema_change.out | 0
.../txn_insert_with_specify_columns.out| 0
...pecify_columns_schema_change_add_key_column.out | 15 +++
...cify_columns_schema_change_add_value_column.out | 15 +++
...pecify_columns_schema_change_reorder_column.out | 19 +++
.../insert_p0/{ => transaction}/test_txn.groovy| 0
.../insert_p0/{ => transaction}/txn_insert.groovy | 6 +-
.../txn_insert_concurrent_insert.groovy| 4 +-
.../txn_insert_inject_case.groovy | 2 +-
.../{ => transaction}/txn_insert_restart_fe.groovy | 0
...txn_insert_restart_fe_with_schema_change.groovy | 0
.../txn_insert_values_with_schema_change.groovy| 2 +-
.../{ => transaction}/txn_insert_with_drop.groovy | 0
.../txn_insert_with_schema_change.groovy | 2 +-
.../txn_insert_with_specify_columns.groovy | 0
...ify_columns_schema_change_add_key_column.groovy | 127
...y_columns_schema_change_add_value_column.groovy | 129 +
...ify_columns_schema_change_reorder_column.groovy | 129 +
34 files changed, 500 insertions(+), 15 deletions(-)
rename regression-test/data/insert_p0/{ => transaction}/test_txn.out (100%)
rename regression-test/data/insert_p0/{ => transaction}/txn_insert.out (100%)
rename regression-test/data/insert_p0/{ =>
transaction}/txn_insert_inject_case.out (100%)
rename regression-test/data/insert_p0/{ =>
transaction}/txn_insert_restart_fe.out (100%)
rename regression-test/data/insert_p0/{ =>
transaction}/txn_insert_restart_fe_with_schema_change.out (100%)
rename regression-test/data/insert_p0/{ =>
transaction}/txn_insert_values_with_schema_change.out (100%)
rename regression-test/data/insert_p0/{ =>
transaction}/txn_insert_with_drop.out (100%)
rename regression-test/data/insert_p0/{ =>
transaction}/txn_insert_with_schema_change.out (100%)
rename regression-test/data/insert_p0/{ =>
transaction}/txn_insert_with_specify_columns.out (100%)
create mode 100644
regression-test/data/insert_p0/transaction/txn_insert_with_specify_columns_schema_change_add_key_column.out
create mode 100644
regression-test/data/insert_p0/transaction/txn_insert_with_specify_columns_schema_change_add_value_column.out
create mode 100644
regression-test/data/insert_p0/transaction/txn_insert_with_specify_columns_schema_change_reorder_column.out
rename regression-test/suites/insert_p0/{ => transaction}/test_txn.groovy
(100%)
rename regression-test/suites/insert_p0/{ => transaction}/txn_insert.groovy
(99%)
rename regression-test/suites/insert_p0/{ =>
transaction}/txn_insert_concurrent_insert.groovy (97%)
rename regression-test/suites/insert_p0/{ =>
transaction}/txn_insert_inject_case.groovy (99%)
rename regression-test/suites/insert_p0/{ =>
transaction}/txn_insert_restart_fe.groovy (100%)
rename regression-test/suites/insert_p0/{ =>
transaction}/txn_insert_restart_fe_with_schema_change.groovy (100%)
rename regression-test/suites/insert_p0/{ =>
transaction}/txn_insert_values_with_schema_change.groovy (98%)
rename regression-test/suites/insert_p0/{ =>
transaction}/txn_insert_with_drop.groovy (100%)
rename regression-test/suites/insert_p0/{ =>
transaction}/txn_insert_with_schema_change.groovy
(doris) 01/02: [Fix](parquet-reader) Fix definition level rle decode dead loop in parquet-reader. (#39523)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit bef810ad6fe4dfd6044305f85786abe7aa3746ef
Author: Qi Chen
AuthorDate: Mon Aug 26 23:26:53 2024 +0800
[Fix](parquet-reader) Fix definition level rle decode dead loop in
parquet-reader. (#39523)
---
be/src/util/bit_stream_utils.h | 4
be/src/util/rle_encoding.h | 2 ++
be/src/vec/exec/format/parquet/level_decoder.h | 4 +++-
.../exec/format/parquet/vparquet_column_reader.cpp | 20 ++--
4 files changed, 27 insertions(+), 3 deletions(-)
diff --git a/be/src/util/bit_stream_utils.h b/be/src/util/bit_stream_utils.h
index 550919440a8..b9b3621cf8b 100644
--- a/be/src/util/bit_stream_utils.h
+++ b/be/src/util/bit_stream_utils.h
@@ -145,6 +145,10 @@ public:
bool is_initialized() const { return buffer_ != nullptr; }
+const uint8_t* buffer() const { return buffer_; }
+
+int max_bytes() const { return max_bytes_; }
+
private:
// Used by SeekToBit() and GetValue() to fetch the
// the next word into buffer_.
diff --git a/be/src/util/rle_encoding.h b/be/src/util/rle_encoding.h
index be4df12916b..206349b4728 100644
--- a/be/src/util/rle_encoding.h
+++ b/be/src/util/rle_encoding.h
@@ -120,6 +120,8 @@ public:
// Get current repeated value, make sure that count equals repeated_count()
T get_repeated_value(size_t count);
+const BitReader& bit_reader() const { return bit_reader_; }
+
private:
bool ReadHeader();
diff --git a/be/src/vec/exec/format/parquet/level_decoder.h
b/be/src/vec/exec/format/parquet/level_decoder.h
index 4f76ac06837..de2f80d7f12 100644
--- a/be/src/vec/exec/format/parquet/level_decoder.h
+++ b/be/src/vec/exec/format/parquet/level_decoder.h
@@ -56,6 +56,8 @@ public:
inline void rewind_one() { _rle_decoder.RewindOne(); }
+const RleDecoder& rle_decoder() const { return _rle_decoder; }
+
private:
tparquet::Encoding::type _encoding;
level_t _bit_width = 0;
@@ -65,4 +67,4 @@ private:
BitReader _bit_packed_decoder;
};
-} // namespace doris::vectorized
\ No newline at end of file
+} // namespace doris::vectorized
diff --git a/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp
b/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp
index c51a51bac3c..c31c63ee87c 100644
--- a/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp
+++ b/be/src/vec/exec/format/parquet/vparquet_column_reader.cpp
@@ -209,7 +209,15 @@ Status ScalarColumnReader::_skip_values(size_t num_values)
{
level_t def_level = -1;
size_t loop_skip = def_decoder.get_next_run(&def_level, num_values
- skipped);
if (loop_skip == 0) {
-continue;
+std::stringstream ss;
+auto& bit_reader = def_decoder.rle_decoder().bit_reader();
+ss << "def_decoder buffer (hex): ";
+for (size_t i = 0; i < bit_reader.max_bytes(); ++i) {
+ss << std::hex << std::setw(2) << std::setfill('0')
+ << static_cast(bit_reader.buffer()[i]) << " ";
+}
+LOG(WARNING) << ss.str();
+return Status::InternalError("Failed to decode definition
level.");
}
if (def_level == 0) {
null_size += loop_skip;
@@ -254,7 +262,15 @@ Status ScalarColumnReader::_read_values(size_t num_values,
ColumnPtr& doris_colu
level_t def_level;
size_t loop_read = def_decoder.get_next_run(&def_level,
num_values - has_read);
if (loop_read == 0) {
-continue;
+std::stringstream ss;
+auto& bit_reader = def_decoder.rle_decoder().bit_reader();
+ss << "def_decoder buffer (hex): ";
+for (size_t i = 0; i < bit_reader.max_bytes(); ++i) {
+ss << std::hex << std::setw(2) << std::setfill('0')
+ << static_cast(bit_reader.buffer()[i]) << " ";
+}
+LOG(WARNING) << ss.str();
+return Status::InternalError("Failed to decode definition
level.");
}
bool is_null = def_level == 0;
if (!(prev_is_null ^ is_null)) {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 02/02: [Enhancement](txn) Block new insert into if schema change happens during transaction (#39483)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f925b6ea70541311faa35dcb5c96e92324debcff
Author: abmdocrt
AuthorDate: Wed Aug 28 23:14:26 2024 +0800
[Enhancement](txn) Block new insert into if schema change happens during
transaction (#39483)
---
be/src/olap/schema_change.cpp | 1 +
.../insert/BatchInsertIntoTableCommand.java| 11 ++
.../trees/plans/physical/PhysicalResultSink.java | 4 -
.../java/org/apache/doris/qe/StmtExecutor.java | 1 +
.../apache/doris/transaction/TransactionEntry.java | 18 +++
.../data/insert_p0/{ => transaction}/test_txn.out | 0
.../insert_p0/{ => transaction}/txn_insert.out | 0
.../{ => transaction}/txn_insert_inject_case.out | 0
.../{ => transaction}/txn_insert_restart_fe.out| 0
.../txn_insert_restart_fe_with_schema_change.out | 0
.../txn_insert_values_with_schema_change.out | 0
.../{ => transaction}/txn_insert_with_drop.out | 0
.../txn_insert_with_schema_change.out | 0
.../txn_insert_with_specify_columns.out| 0
...pecify_columns_schema_change_add_key_column.out | 15 +++
...cify_columns_schema_change_add_value_column.out | 15 +++
...pecify_columns_schema_change_reorder_column.out | 19 +++
.../insert_p0/{ => transaction}/test_txn.groovy| 0
.../insert_p0/{ => transaction}/txn_insert.groovy | 6 +-
.../txn_insert_concurrent_insert.groovy| 4 +-
.../txn_insert_inject_case.groovy | 2 +-
.../{ => transaction}/txn_insert_restart_fe.groovy | 0
...txn_insert_restart_fe_with_schema_change.groovy | 0
.../txn_insert_values_with_schema_change.groovy| 2 +-
.../{ => transaction}/txn_insert_with_drop.groovy | 0
.../txn_insert_with_schema_change.groovy | 2 +-
.../txn_insert_with_specify_columns.groovy | 0
...ify_columns_schema_change_add_key_column.groovy | 127
...y_columns_schema_change_add_value_column.groovy | 129 +
...ify_columns_schema_change_reorder_column.groovy | 129 +
30 files changed, 473 insertions(+), 12 deletions(-)
diff --git a/be/src/olap/schema_change.cpp b/be/src/olap/schema_change.cpp
index 04409e37d84..4418d02ecdf 100644
--- a/be/src/olap/schema_change.cpp
+++ b/be/src/olap/schema_change.cpp
@@ -777,6 +777,7 @@ SchemaChangeJob::SchemaChangeJob(StorageEngine&
local_storage_engine,
// The admin should upgrade all BE and then upgrade FE.
// Should delete the old code after upgrade finished.
Status SchemaChangeJob::_do_process_alter_tablet(const TAlterTabletReqV2&
request) {
+DBUG_EXECUTE_IF("SchemaChangeJob._do_process_alter_tablet.sleep", {
sleep(10); })
Status res;
signal::tablet_id = _base_tablet->get_table_id();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BatchInsertIntoTableCommand.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BatchInsertIntoTableCommand.java
index a1b834fe904..04d43e8cc33 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BatchInsertIntoTableCommand.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BatchInsertIntoTableCommand.java
@@ -20,6 +20,7 @@ package org.apache.doris.nereids.trees.plans.commands.insert;
import org.apache.doris.analysis.StmtType;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.Table;
import org.apache.doris.catalog.TableIf;
import org.apache.doris.common.ErrorCode;
@@ -120,6 +121,16 @@ public class BatchInsertIntoTableCommand extends Command
implements NoForward, E
Preconditions.checkArgument(plan.isPresent(), "insert into command
must contain OlapTableSinkNode");
sink = ((PhysicalOlapTableSink) plan.get());
Table targetTable = sink.getTargetTable();
+if (ctx.getTxnEntry().isFirstTxnInsert()) {
+ctx.getTxnEntry().setTxnSchemaVersion(((OlapTable)
targetTable).getBaseSchemaVersion());
+ctx.getTxnEntry().setFirstTxnInsert(false);
+} else {
+if (((OlapTable) targetTable).getBaseSchemaVersion() !=
ctx.getTxnEntry().getTxnSchemaVersion()) {
+throw new AnalysisException("There are schema changes in
one transaction, "
++ "you can commit this transaction with formal
data or rollback "
++ "this whole transaction.");
+}
+}
// should set columns of sink since we maybe generate some
invisible columns
List fullSchema = sink.getTargetTable().getFullSchema();
List targetSchema = Lists.n
Re: [PR] [Feature](compaction) Support full compaction by table id for cloud [doris]
doris-robot commented on PR #39934: URL: https://github.com/apache/doris/pull/39934#issuecomment-2325844738 TPC-DS: Total hot run time: 192895 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 9807c0aed409ecb5f6cb80d67d24a8ac30986249, data reload: false query1 1263881 855 855 query2 6342200519781978 query3 10637 399240283992 query4 59774 26457 23224 23224 query5 5412510 489 489 query6 404 163 159 159 query7 5773298 288 288 query8 293 203 219 203 query9 8968246424622462 query10 483 279 266 266 query11 16729 15170 15430 15170 query12 163 107 103 103 query13 1548395 380 380 query14 10668 735472607260 query15 227 180 202 180 query16 7393490 467 467 query17 1108589 639 589 query18 1600292 297 292 query19 270 154 145 145 query20 129 114 112 112 query21 200 102 104 102 query22 4647441546854415 query23 34338 33441 33548 33441 query24 5969287928312831 query25 548 403 405 403 query26 670 156 155 155 query27 1753284 277 277 query28 3590204120302030 query29 667 445 401 401 query30 235 154 154 154 query31 909 742 775 742 query32 81 49 53 49 query33 453 285 271 271 query34 851 479 481 479 query35 848 731 748 731 query36 1084955 930 930 query37 145 94 87 87 query38 3999400738793879 query39 1451141314301413 query40 198 117 112 112 query41 44 47 44 44 query42 112 96 94 94 query43 532 485 492 485 query44 1122726 736 726 query45 193 164 165 164 query46 1092733 737 733 query47 1906177718331777 query48 377 292 291 291 query49 757 425 436 425 query50 834 423 418 418 query51 7010689968866886 query52 100 90 86 86 query53 250 175 180 175 query54 572 459 455 455 query55 76 76 75 75 query56 277 257 266 257 query57 1156109310641064 query58 232 245 244 244 query59 3012293729652937 query60 294 282 266 266 query61 103 101 105 101 query62 750 647 642 642 query63 218 181 189 181 query64 2382686 654 654 query65 3230312831903128 query66 679 343 383 343 query67 15541 15242 15168 15168 query68 3011552 564 552 query69 401 291 271 271 query70 1177115911011101 query71 343 278 290 278 query72 5411405439873987 query73 751 327 329 327 query74 9348889888948894 query75 3366260726792607 query76 14231032998 998 query77 520 338 349 338 query78 9789937290059005 query79 1048532 526 526 query80 702 510 501 501 query81 539 237 229 229 query82 254 145 148 145 query83 175 149 151 149 query84 266 76 76 76 query85 669 287 343 287 query86 298 270 303 270 query87 4433426342824263 query88 3306232823312328 query89 369 295 274 274 query90 1743196 191 191 query91 135 104 106 104 query92 61 51 51 51 query93 1045528 522 522 query94 556 297 300 297 query95 336 254 261 254 query96 593 270 260 260 query97 3210306930253025 query98 219 201 197 197 query99 1600130412531253 Total cold run time: 301768 ms Total hot run time: 192895 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improvement](tablet report) tablet report increase report version [doris]
doris-robot commented on PR #40172: URL: https://github.com/apache/doris/pull/40172#issuecomment-2325847185 TPC-H: Total hot run time: 37888 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit d171035d216f3dd2e5d41a5817363648eaa86aa7, data reload: false -- Round 1 -- q1 17637 436544604365 q2 2036187 174 174 q3 11777 995 1020995 q4 10511 680 753 680 q5 7736285127292729 q6 224 140 143 140 q7 969 615 620 615 q8 9328204020482040 q9 7744651665066506 q10 6987219022022190 q11 448 238 235 235 q12 406 228 227 227 q13 18908 306030953060 q14 276 232 228 228 q15 507 480 491 480 q16 582 522 513 513 q17 983 674 704 674 q18 7299680468796804 q19 1385106610991066 q20 672 336 337 336 q21 4118283629772836 q22 11101008995 995 Total cold run time: 111643 ms Total hot run time: 37888 ms - Round 2, with runtime_filter_mode=off - q1 4372430542604260 q2 388 277 262 262 q3 2912261926662619 q4 1925168316231623 q5 5640565256975652 q6 227 148 141 141 q7 2244184217771777 q8 3334346634683466 q9 8884881687688768 q10 3589342133453345 q11 604 512 507 507 q12 833 673 644 644 q13 12655 322932943229 q14 334 287 294 287 q15 523 493 513 493 q16 621 604 593 593 q17 1842156415391539 q18 8084779678257796 q19 1739164815541554 q20 2167188619551886 q21 5659554153935393 q22 1135105711211057 Total cold run time: 69711 ms Total hot run time: 56891 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ES Catalog)Do not extract doc_values of field with ignore_above setting [doris]
doris-robot commented on PR #40314: URL: https://github.com/apache/doris/pull/40314#issuecomment-2325847350 TPC-H: Total hot run time: 38498 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b8336187df0ca4505775aba9d493eb5a88d74f73, data reload: false -- Round 1 -- q1 17754 439543494349 q2 2018192 177 177 q3 11639 970 1085970 q4 10512 787 779 779 q5 7734285728112811 q6 231 148 148 148 q7 967 621 614 614 q8 9345208420872084 q9 7095653765646537 q10 7005226722432243 q11 451 260 251 251 q12 400 234 234 234 q13 17766 312830963096 q14 273 250 232 232 q15 526 500 500 500 q16 585 523 507 507 q17 988 715 677 677 q18 7272677369086773 q19 1417104710701047 q20 698 336 333 333 q21 3994313131233123 q22 1095102310131013 Total cold run time: 109765 ms Total hot run time: 38498 ms - Round 2, with runtime_filter_mode=off - q1 4414435143474347 q2 378 270 272 270 q3 2887259926402599 q4 1915163116391631 q5 5672572557295725 q6 239 139 141 139 q7 2302186318581858 q8 3398348335543483 q9 8845893489848934 q10 3535343134243424 q11 610 545 515 515 q12 815 659 701 659 q13 13631 337432553255 q14 319 287 299 287 q15 534 495 496 495 q16 598 593 594 593 q17 1884159315391539 q18 8191773678967736 q19 1729147916201479 q20 2159192919441929 q21 5914552955815529 q22 1130103010211021 Total cold run time: 71099 ms Total hot run time: 57447 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](aes_encrypt) support GCM mode for aes_encrypt and aes_decrypt [doris]
cambyzju commented on PR #40004: URL: https://github.com/apache/doris/pull/40004#issuecomment-2325849353 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](DCHECK) Fix the DCHECK failure when the query is canceled. [doris]
doris-robot commented on PR #40276: URL: https://github.com/apache/doris/pull/40276#issuecomment-2325849998 TPC-DS: Total hot run time: 192684 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 97c9c4e6343572e40990a7fafae768c63fdb7d62, data reload: false query1 1261879 861 861 query2 6310199719621962 query3 10681 415137853785 query4 60093 26338 23425 23425 query5 5452485 499 485 query6 411 166 156 156 query7 5843287 287 287 query8 279 201 202 201 query9 8722244224192419 query10 465 269 254 254 query11 17841 15025 15428 15025 query12 151 96 100 96 query13 1547400 390 390 query14 11194 732270357035 query15 261 168 189 168 query16 7582466 495 466 query17 1106579 561 561 query18 1965302 283 283 query19 295 151 143 143 query20 118 109 110 109 query21 207 112 103 103 query22 4530445143724372 query23 34352 33585 33811 33585 query24 5911284028022802 query25 527 378 375 375 query26 686 149 146 146 query27 1800271 275 271 query28 3860203820112011 query29 692 401 408 401 query30 235 150 160 150 query31 948 762 772 762 query32 76 52 54 52 query33 435 279 273 273 query34 866 476 494 476 query35 847 723 706 706 query36 1065918 943 918 query37 154 87 89 87 query38 4009397439073907 query39 1430144213961396 query40 210 115 113 113 query41 47 45 45 45 query42 111 92 93 92 query43 520 485 491 485 query44 1096719 727 719 query45 193 164 164 164 query46 1089760 738 738 query47 1934176318261763 query48 370 304 290 290 query49 779 451 445 445 query50 817 422 423 422 query51 6887688068256825 query52 103 86 87 86 query53 251 182 180 180 query54 568 455 460 455 query55 79 76 74 74 query56 278 268 277 268 query57 1193107110731071 query58 238 235 230 230 query59 3090284127942794 query60 305 270 317 270 query61 102 103 97 97 query62 754 658 654 654 query63 222 180 183 180 query64 2851668 673 668 query65 3295315731393139 query66 627 359 347 347 query67 15437 15336 15259 15259 query68 2962563 576 563 query69 401 282 271 271 query70 1171107211321072 query71 333 267 277 267 query72 6083416540434043 query73 778 333 330 330 query74 9201885488378837 query75 3353262926982629 query76 1361999 966 966 query77 510 318 316 316 query78 10839 926293319262 query79 1042523 520 520 query80 704 499 504 499 query81 468 236 240 236 query82 248 145 145 145 query83 171 144 146 144 query84 251 126 77 77 query85 690 298 284 284 query86 314 272 303 272 query87 4409436943674367 query88 3158231623342316 query89 375 281 294 281 query90 2016187 188 187 query91 126 104 103 103 query92 58 50 48 48 query93 1061522 523 522 query94 680 305 297 297 query95 336 255 260 255 query96 585 259 272 259 query97 3187309230523052 query98 218 200 212 200 query99 1628128712881287 Total cold run time: 306370 ms Total hot run time: 192684 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t
Re: [PR] [fix](nereids) fix bug for A>n, where A.max is infinity [doris]
doris-robot commented on PR #39936: URL: https://github.com/apache/doris/pull/39936#issuecomment-2325853197 TPC-DS: Total hot run time: 186664 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 0c671a08b6b4de706f5be7dd3b5521f51f478275, data reload: false query1 922 374 364 364 query2 6474205319721972 query3 6652226 214 214 query4 33709 23004 23153 23004 query5 4197499 492 492 query6 252 160 171 160 query7 4569298 298 298 query8 267 216 219 216 query9 8631246624302430 query10 434 275 269 269 query11 17678 15084 14868 14868 query12 144 95 96 95 query13 1648392 376 376 query14 9398636373616363 query15 250 170 175 170 query16 7539454 473 454 query17 1646585 564 564 query18 2016300 287 287 query19 337 159 149 149 query20 122 113 111 111 query21 212 112 104 104 query22 4473402442864024 query23 34192 33656 34005 33656 query24 11224 286328212821 query25 668 406 395 395 query26 1572158 156 156 query27 2724280 279 279 query28 7242206120172017 query29 911 433 427 427 query30 289 154 150 150 query31 981 737 768 737 query32 106 57 58 57 query33 763 296 297 296 query34 943 462 476 462 query35 862 720 712 712 query36 1086918 955 918 query37 153 90 98 90 query38 3986392538103810 query39 1461140914341409 query40 267 116 115 115 query41 52 45 46 45 query42 117 93 93 93 query43 517 503 469 469 query44 1199753 724 724 query45 196 167 168 167 query46 1103754 756 754 query47 1886181017741774 query48 381 300 302 300 query49 1100423 418 418 query50 808 415 416 415 query51 6978693268536853 query52 98 88 86 86 query53 250 183 177 177 query54 978 480 456 456 query55 72 75 77 75 query56 274 261 272 261 query57 1191105710591057 query58 235 249 232 232 query59 3115275428402754 query60 300 266 271 266 query61 108 99 98 98 query62 816 659 675 659 query63 215 188 178 178 query64 5204678 648 648 query65 3222314431283128 query66 1251339 364 339 query67 15994 15185 15380 15185 query68 4477556 555 555 query69 402 275 283 275 query70 1216116911791169 query71 344 284 277 277 query72 6551402040294020 query73 740 318 326 318 query74 9278881489098814 query75 3430269126402640 query76 2584107410201020 query77 444 326 303 303 query78 9698903595649035 query79 1315522 523 522 query80 923 506 486 486 query81 575 235 234 234 query82 1276145 142 142 query83 273 155 155 155 query84 235 81 86 81 query85 1011293 274 274 query86 301 273 287 273 query87 4463420042374200 query88 2977234023262326 query89 386 303 288 288 query90 1676194 191 191 query91 126 103 104 103 query92 58 52 48 48 query93 1046528 526 526 query94 553 292 320 292 query95 340 249 282 249 query96 594 262 263 262 query97 3220304130793041 query98 219 199 190 190 query99 1542125312681253 Total cold run time: 290933 ms Total hot run time: 186664 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t
