[GitHub] [doris] github-actions[bot] commented on pull request #22382: [feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load
github-actions[bot] commented on PR #22382: URL: https://github.com/apache/doris/pull/22382#issuecomment-1657783565 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zddr commented on a diff in pull request #22272: [fix](rest)check response code when get image
zddr commented on code in PR #22272:
URL: https://github.com/apache/doris/pull/22272#discussion_r1278866052
##
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##
@@ -1642,13 +1640,19 @@ private void getNewImage(HostInfo helperNode) throws
IOException {
try {
String hostPort =
NetUtils.getHostPortInAccessibleFormat(helperNode.getHost(), Config.http_port);
String infoUrl = "http://"; + hostPort + "/info";
-StorageInfo info = getStorageInfo(infoUrl);
+ResponseBody responseBody = MetaHelper
+.doGet(infoUrl, HTTP_TIMEOUT_SECOND * 1000,
StorageInfo.class);
Review Comment:
ResponseBody Equivalent to StorageInfoV2,so we do not need
StorageInfoV2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] LiBinfeng-01 commented on pull request #22196: [Fix](Planner) fix parse error of view with group_concat order by
LiBinfeng-01 commented on PR #22196: URL: https://github.com/apache/doris/pull/22196#issuecomment-1657786914 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #22211: [enhancement](nereids) Execute sync analyze task with multi-thread
morrySnow merged PR #22211: URL: https://github.com/apache/doris/pull/22211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [enhancement](nereids) Execute sync analyze task with multi-thread (#22211)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 4c6458aa77 [enhancement](nereids) Execute sync analyze task with
multi-thread (#22211)
4c6458aa77 is described below
commit 4c6458aa778a94d27d15ee579803c9aec584fa31
Author: AKIRA <33112463+kikyou1...@users.noreply.github.com>
AuthorDate: Mon Jul 31 15:05:07 2023 +0800
[enhancement](nereids) Execute sync analyze task with multi-thread (#22211)
It was executed in sequentialy, which may cause a lot of time
---
.../apache/doris/statistics/AnalysisManager.java | 64 +++---
1 file changed, 45 insertions(+), 19 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
index cb6a3dfe5c..66f0b94aa8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
@@ -41,6 +41,7 @@ import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.FeConstants;
+import org.apache.doris.common.ThreadPoolManager.BlockedPolicy;
import org.apache.doris.common.io.Writable;
import org.apache.doris.common.util.Daemon;
import org.apache.doris.common.util.Util;
@@ -59,6 +60,7 @@ import org.apache.doris.statistics.util.StatisticsUtil;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Maps;
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.jetbrains.annotations.Nullable;
@@ -82,6 +84,9 @@ import java.util.StringJoiner;
import java.util.TreeMap;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentMap;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.SynchronousQueue;
+import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import java.util.function.Function;
import java.util.function.Predicate;
@@ -738,12 +743,23 @@ public class AnalysisManager extends Daemon implements
Writable {
ConnectContext ctx = ConnectContext.get();
try {
ctxToSyncTask.put(ctx, syncTaskCollection);
-syncTaskCollection.execute();
+ThreadPoolExecutor syncExecPool = createThreadPoolForSyncAnalyze();
+syncTaskCollection.execute(syncExecPool);
} finally {
ctxToSyncTask.remove(ctx);
}
}
+private ThreadPoolExecutor createThreadPoolForSyncAnalyze() {
+String poolName = "SYNC ANALYZE THREAD POOL";
+return new ThreadPoolExecutor(0, 64,
+0, TimeUnit.SECONDS,
+new SynchronousQueue(),
+new ThreadFactoryBuilder().setDaemon(true).setNameFormat("SYNC
ANALYZE" + "-%d")
+.build(), new BlockedPolicy(poolName,
+(int)
TimeUnit.HOURS.toSeconds(Config.analyze_task_timeout_in_hours)));
+}
+
public void dropStats(DropStatsStmt dropStatsStmt) throws DdlException {
if (dropStatsStmt.dropExpired) {
Env.getCurrentEnv().getStatisticsCleaner().clear();
@@ -844,28 +860,38 @@ public class AnalysisManager extends Daemon implements
Writable {
tasks.forEach(BaseAnalysisTask::cancel);
}
-public void execute() {
-List colNames = new ArrayList<>();
-List errorMessages = new ArrayList<>();
+public void execute(ThreadPoolExecutor executor) {
+List colNames = Collections.synchronizedList(new
ArrayList<>());
+List errorMessages = Collections.synchronizedList(new
ArrayList<>());
+CountDownLatch countDownLatch = new CountDownLatch(tasks.size());
for (BaseAnalysisTask task : tasks) {
-if (cancelled) {
-colNames.add(task.info.colName);
-errorMessages.add("Cancelled");
-continue;
-}
-try {
-task.doExecute();
-updateSyncTaskStatus(task, AnalysisState.FINISHED);
-} catch (Throwable t) {
-colNames.add(task.info.colName);
-errorMessages.add(Util.getRootCauseMessage(t));
-updateSyncTaskStatus(task, AnalysisState.FAILED);
-LOG.warn("Failed to analyze, info: {}", task, t);
-}
+executor.submit(() -> {
+try {
+if (cancelled) {
+return;
+}
+
[GitHub] [doris] github-actions[bot] commented on pull request #21970: [enhancement](nereids) Improve stats preload performance
github-actions[bot] commented on PR #21970: URL: https://github.com/apache/doris/pull/21970#issuecomment-1657788026 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #21970: [enhancement](nereids) Improve stats preload performance
github-actions[bot] commented on PR #21970: URL: https://github.com/apache/doris/pull/21970#issuecomment-1657788077 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] zddr commented on pull request #22272: [fix](rest)check response code when get image
zddr commented on PR #22272: URL: https://github.com/apache/doris/pull/22272#issuecomment-1657793402 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22404: [fix][regression-test] change lazy open regression test name
github-actions[bot] commented on PR #22404: URL: https://github.com/apache/doris/pull/22404#issuecomment-1657794963 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22314: [feature](property) Add table property "is_being_synced"
github-actions[bot] commented on PR #22314: URL: https://github.com/apache/doris/pull/22314#issuecomment-1657797156 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #22364: [FIX](decimal)fix decimal precision
yiguolei commented on code in PR #22364: URL: https://github.com/apache/doris/pull/22364#discussion_r1278879260 ## be/src/util/string_parser.hpp: ## @@ -624,27 +624,29 @@ T StringParser::string_to_decimal(const char* s, int len, int type_precision, in } Review Comment: decimalv2(18,3) integer: 15, float:3 1234567890123456.111 ==> 1234567890123456.111 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22317: [improvement](tablet clone) improve scaling out speed
hello-stephen commented on PR #22317: URL: https://github.com/apache/doris/pull/22317#issuecomment-1657811290 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.25 seconds stream load tsv: 511 seconds loaded 74807831229 Bytes, about 139 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 30.1 seconds inserted 1000 Rows, about 332K ops/s storage size: 17162247698 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22404: [fix][regression-test] change lazy open regression test name
hello-stephen commented on PR #22404: URL: https://github.com/apache/doris/pull/22404#issuecomment-1657810484 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.35 seconds stream load tsv: 506 seconds loaded 74807831229 Bytes, about 140 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.0 seconds inserted 1000 Rows, about 344K ops/s storage size: 17166601015 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #22327: [Improvement](aggregation) optimization for aggregation hash_table_lazy_emplace
HappenLee commented on code in PR #22327:
URL: https://github.com/apache/doris/pull/22327#discussion_r127604
##
be/src/vec/common/aggregation_common.h:
##
@@ -218,6 +196,70 @@ T pack_fixed(size_t i, size_t keys_size, const
ColumnRawPtrs& key_columns, const
return key;
}
+template
+std::vector pack_fixeds(size_t row_numbers, const ColumnRawPtrs&
key_columns,
+ const Sizes& key_sizes, const ColumnRawPtrs&
nullmap_columns) {
+size_t bitmap_size = 0;
+if (!nullmap_columns.empty()) {
+bitmap_size = std::tuple_size>::value;
+}
+
+std::vector result(row_numbers);
+size_t offset = 0;
+if (bitmap_size > 0) {
+for (size_t j = 0; j < nullmap_columns.size(); j++) {
+if (!nullmap_columns[j]) {
+continue;
+}
+size_t bucket = j / 8;
+size_t offset = j % 8;
+const auto& data =
+assert_cast(*nullmap_columns[j]).get_data().data();
+for (size_t i = 0; i < row_numbers; ++i) {
+*((char*)(&result[i]) + bucket) |= data[i] << offset;
+}
+}
+offset += bitmap_size;
+}
+
+for (size_t j = 0; j < key_columns.size(); ++j) {
+const char* data = key_columns[j]->get_raw_data().data;
+
+auto foo = [&](Fixed zero) {
+if (nullmap_columns.size() && nullmap_columns[j]) {
+const auto& nullmap =
+assert_cast(*nullmap_columns[j]).get_data().data();
+for (size_t i = 0; i < row_numbers; ++i) {
+// make sure null cell is filled by 0x0
+memcpy_fixed((char*)(&result[i]) + offset,
+nullmap[i] ? (char*)&zero : data + i *
key_sizes[j]);
Review Comment:
`key_sizes[j]` replace by `sizeof(Fixed)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #22406: [opt](conf) Modify brpc work pool conf default value
xinyiZzz opened a new pull request, #22406: URL: https://github.com/apache/doris/pull/22406 ## Proposed changes Default, if less than or equal 32 core, the following are 128, 128, 10240, 10240 in turn. if greater than 32 core, the following are core num * 4, core num * 4, core num * 320, core num * 320 in turn 1. brpc_heavy_work_pool_threads 2. brpc_light_work_pool_threads 3. brpc_heavy_work_pool_max_queue_size 4. brpc_light_work_pool_max_queue_size ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22264: [feature](Nereids): pushdown MIN/MAX/SUM through join
hello-stephen commented on PR #22264: URL: https://github.com/apache/doris/pull/22264#issuecomment-1657833224 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.45 seconds stream load tsv: 509 seconds loaded 74807831229 Bytes, about 140 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 69 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.4 seconds inserted 1000 Rows, about 340K ops/s storage size: 17162506728 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22406: [opt](conf) Modify brpc work pool conf default value
github-actions[bot] commented on PR #22406: URL: https://github.com/apache/doris/pull/22406#issuecomment-1657836187 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22406: [opt](conf) Modify brpc work pool conf default value
github-actions[bot] commented on PR #22406: URL: https://github.com/apache/doris/pull/22406#issuecomment-1657836238 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #22327: [Improvement](aggregation) optimization for aggregation hash_table_lazy_emplace
BiteThet commented on PR #22327: URL: https://github.com/apache/doris/pull/22327#issuecomment-1657837884 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22406: [opt](conf) Modify brpc work pool conf default value
github-actions[bot] commented on PR #22406: URL: https://github.com/apache/doris/pull/22406#issuecomment-1657839280 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] sohardforaname commented on pull request #22195: [Feature](Nereids)support create table and CTAS for Nereids
sohardforaname commented on PR #22195: URL: https://github.com/apache/doris/pull/22195#issuecomment-1657843954 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22327: [Improvement](aggregation) optimization for aggregation hash_table_lazy_emplace
github-actions[bot] commented on PR #22327: URL: https://github.com/apache/doris/pull/22327#issuecomment-1657846798 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22272: [fix](rest)check response code when get image
hello-stephen commented on PR #22272: URL: https://github.com/apache/doris/pull/22272#issuecomment-1657848935 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 48.21 seconds stream load tsv: 509 seconds loaded 74807831229 Bytes, about 140 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.4 seconds inserted 1000 Rows, about 340K ops/s storage size: 17162097084 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] KassieZ commented on pull request #22312: [Docs](Community) Update Join Community Invalid Pics
KassieZ commented on PR #22312: URL: https://github.com/apache/doris/pull/22312#issuecomment-1657853641 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22341: [typo](docs) fix array_intersect function sample error
github-actions[bot] commented on PR #22341: URL: https://github.com/apache/doris/pull/22341#issuecomment-1657861264 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-spark-connector] gnehil opened a new pull request, #123: [fix] json format load default mode
gnehil opened a new pull request, #123: URL: https://github.com/apache/doris-spark-connector/pull/123 # Proposed changes Issue Number: close #xxx ## Problem Summary: Fixed the problem that there is no default parameter when `read_json_by_line` and `strip_outer_array` are not specified in json format. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #22406: [opt](conf) Modify brpc work pool conf default value
xinyiZzz commented on PR #22406: URL: https://github.com/apache/doris/pull/22406#issuecomment-1657866168 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz commented on pull request #22407: [opt](conf) Modify brpc eovercrowded conf
xinyiZzz commented on PR #22407: URL: https://github.com/apache/doris/pull/22407#issuecomment-1657869842 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #22407: [opt](conf) Modify brpc eovercrowded conf
xinyiZzz opened a new pull request, #22407: URL: https://github.com/apache/doris/pull/22407 ## Proposed changes 1. brpc ignore eovercrowded of data stream sender and exchange sink buffer 2. Modify the default value of brpc_socket_max_unwritten_bytes ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22331: [fix](complex_type) throw error when reading complex types in broker/stream load
github-actions[bot] commented on PR #22331: URL: https://github.com/apache/doris/pull/22331#issuecomment-1657874764 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on pull request #22327: [Improvement](aggregation) optimization for aggregation hash_table_lazy_emplace
BiteThet commented on PR #22327: URL: https://github.com/apache/doris/pull/22327#issuecomment-1657879949 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22407: [opt](conf) Modify brpc eovercrowded conf
github-actions[bot] commented on PR #22407: URL: https://github.com/apache/doris/pull/22407#issuecomment-1657881344 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22196: [Fix](Planner) fix parse error of view with group_concat order by
github-actions[bot] commented on PR #22196: URL: https://github.com/apache/doris/pull/22196#issuecomment-1657881623 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ixzc closed pull request #22094: [typo][doc]Modify word typos.
ixzc closed pull request #22094: [typo][doc]Modify word typos. URL: https://github.com/apache/doris/pull/22094 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Jibing-Li closed pull request #22116: [Fix](nereids) Fix Olap table qualifier only contains db name bug.
Jibing-Li closed pull request #22116: [Fix](nereids) Fix Olap table qualifier only contains db name bug. URL: https://github.com/apache/doris/pull/22116 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Jibing-Li opened a new pull request, #22408: [Fix](nereids) Fix Olap table qualifier only contains db name bug.
Jibing-Li opened a new pull request, #22408: URL: https://github.com/apache/doris/pull/22408 While creating LogicalOlapScan, the qualifier only includes database name (without catalog name). This will cause getDatabase function in LogicalCatalogRelation fail to find the correct database, because without catalog name, it only search current catalog. This will cause an error because current catalog may be an external catalog, and Olap database only exist in internal catalog. This pr pass the whole qualifier to LogicalOlapScan, including catalog name, db name and table name. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Jibing-Li commented on pull request #22408: [Fix](nereids) Fix Olap table qualifier only contains db name bug.
Jibing-Li commented on PR #22408: URL: https://github.com/apache/doris/pull/22408#issuecomment-1657885124 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ixzc closed pull request #22085: [typo][doc]Modify the wrong sql description in show-data.
ixzc closed pull request #22085: [typo][doc]Modify the wrong sql description in show-data. URL: https://github.com/apache/doris/pull/22085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] CalvinKirs commented on pull request #21916: [Feature](Job)Support scheduler job
CalvinKirs commented on PR #21916: URL: https://github.com/apache/doris/pull/21916#issuecomment-1657892283 Run FE UT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ixzc closed pull request #22084: [typo][doc]Modify the incorrect sql description in show-partition-id.
ixzc closed pull request #22084: [typo][doc]Modify the incorrect sql description in show-partition-id. URL: https://github.com/apache/doris/pull/22084 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] CalvinKirs commented on pull request #21916: [Feature](Job)Support scheduler job
CalvinKirs commented on PR #21916: URL: https://github.com/apache/doris/pull/21916#issuecomment-1657895698 run P0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22327: [Improvement](aggregation) optimization for aggregation hash_table_lazy_emplace
github-actions[bot] commented on PR #22327: URL: https://github.com/apache/doris/pull/22327#issuecomment-1657898428 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22327: [Improvement](aggregation) optimization for aggregation hash_table_lazy_emplace
github-actions[bot] commented on PR #22327: URL: https://github.com/apache/doris/pull/22327#issuecomment-1657898501 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ixzc closed pull request #21740: [Docs](statistics)Modify the syntax description in the statistics.
ixzc closed pull request #21740: [Docs](statistics)Modify the syntax description in the statistics. URL: https://github.com/apache/doris/pull/21740 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] CalvinKirs commented on pull request #21916: [Feature](Job)Support scheduler job
CalvinKirs commented on PR #21916: URL: https://github.com/apache/doris/pull/21916#issuecomment-1657900202 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ixzc closed pull request #21874: [typo](docs)Add some description in example.
ixzc closed pull request #21874: [typo](docs)Add some description in example. URL: https://github.com/apache/doris/pull/21874 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ixzc closed pull request #21967: [typo][doc]Delete the extra "\".
ixzc closed pull request #21967: [typo][doc]Delete the extra "\". URL: https://github.com/apache/doris/pull/21967 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ixzc closed pull request #21987: [typo][doc]Modify the error description for creating the materialized view syntax.
ixzc closed pull request #21987: [typo][doc]Modify the error description for creating the materialized view syntax. URL: https://github.com/apache/doris/pull/21987 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris-spark-connector] JNSimba merged pull request #123: [fix] json format load default mode
JNSimba merged PR #123: URL: https://github.com/apache/doris-spark-connector/pull/123 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris-spark-connector] branch master updated: fix json mode default value (#123)
This is an automated email from the ASF dual-hosted git repository.
diwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-spark-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 8caab6f fix json mode default value (#123)
8caab6f is described below
commit 8caab6f9d4e3cb2c0efae249316ecd1a8eeeb32d
Author: gnehil
AuthorDate: Mon Jul 31 16:26:37 2023 +0800
fix json mode default value (#123)
---
.../src/main/java/org/apache/doris/spark/load/DorisStreamLoad.java | 3 +++
1 file changed, 3 insertions(+)
diff --git
a/spark-doris-connector/src/main/java/org/apache/doris/spark/load/DorisStreamLoad.java
b/spark-doris-connector/src/main/java/org/apache/doris/spark/load/DorisStreamLoad.java
index c40420d..3708c55 100644
---
a/spark-doris-connector/src/main/java/org/apache/doris/spark/load/DorisStreamLoad.java
+++
b/spark-doris-connector/src/main/java/org/apache/doris/spark/load/DorisStreamLoad.java
@@ -114,6 +114,9 @@ public class DorisStreamLoad implements Serializable {
boolean stripOuterArray =
Boolean.parseBoolean(streamLoadProp.getOrDefault("strip_outer_array", "false"));
if (readJsonByLine && stripOuterArray) {
throw new IllegalArgumentException("Only one of options
'read_json_by_line' and 'strip_outer_array' can be set to true");
+} else if (!readJsonByLine && !stripOuterArray) {
+LOG.info("set default json mode: strip_outer_array");
+streamLoadProp.put("strip_outer_array", "true");
}
}
LINE_DELIMITER =
escapeString(streamLoadProp.getOrDefault("line_delimiter", "\n"));
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ixzc closed pull request #22335: [typo][doc] modify some doc errors.
ixzc closed pull request #22335: [typo][doc] modify some doc errors. URL: https://github.com/apache/doris/pull/22335 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] jeffreys-cat opened a new pull request, #22409: [feature](ui) add profile download button in the list page
jeffreys-cat opened a new pull request, #22409: URL: https://github.com/apache/doris/pull/22409 ## Proposed changes **add profile download button in the list page**  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] CalvinKirs commented on pull request #21916: [Feature](Job)Support scheduler job
CalvinKirs commented on PR #21916: URL: https://github.com/apache/doris/pull/21916#issuecomment-1657934982 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xinyiZzz opened a new pull request, #22410: [opt](conf) scanner thread num is changed to logical disks num * 48
xinyiZzz opened a new pull request, #22410: URL: https://github.com/apache/doris/pull/22410 ## Proposed changes The default value of scanner thread num is changed to logical disks num * 48 ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #22327: [Improvement](aggregation) optimization for aggregation hash_table_lazy_emplace
HappenLee commented on code in PR #22327:
URL: https://github.com/apache/doris/pull/22327#discussion_r1278975397
##
be/src/vec/common/columns_hashing.h:
##
@@ -73,6 +75,14 @@ struct HashMethodOneNumber : public
columns_hashing_impl::HashMethodBase<
FieldType get_key_holder(size_t row, Arena&) const {
return unaligned_load(vec + row * sizeof(FieldType));
}
+
+std::vector get_keys(size_t rows_number) const {
Review Comment:
std::span
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 opened a new pull request, #22411: [fix](string-column) fix unescape result length error
TangSiyang2001 opened a new pull request, #22411: URL: https://github.com/apache/doris/pull/22411 ## Proposed changes Found that text convertor got error result while doing unescape, due to result length error while working on #22382. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on pull request #22409: [feature](ui) add profile download button in the list page
yiguolei commented on PR #22409: URL: https://github.com/apache/doris/pull/22409#issuecomment-1657939105 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #22411: [fix](string-column) fix unescape result length error
TangSiyang2001 commented on PR #22411: URL: https://github.com/apache/doris/pull/22411#issuecomment-1657940579 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22327: [Improvement](aggregation) optimization for aggregation hash_table_lazy_emplace
hello-stephen commented on PR #22327: URL: https://github.com/apache/doris/pull/22327#issuecomment-1657940749 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.02 seconds stream load tsv: 512 seconds loaded 74807831229 Bytes, about 139 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.2 seconds inserted 1000 Rows, about 342K ops/s storage size: 17162063165 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22411: [fix](string-column) fix unescape result length error
github-actions[bot] commented on PR #22411: URL: https://github.com/apache/doris/pull/22411#issuecomment-1657951085 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22410: [opt](conf) scanner thread num is changed to logical disks num * 48
github-actions[bot] commented on PR #22410: URL: https://github.com/apache/doris/pull/22410#issuecomment-1657951770 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #22382: [feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load
TangSiyang2001 commented on PR #22382: URL: https://github.com/apache/doris/pull/22382#issuecomment-1657958909 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ixzc opened a new pull request, #22413: [typo][doc]Modify some sql syntax description errors.
ixzc opened a new pull request, #22413: URL: https://github.com/apache/doris/pull/22413 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangbo opened a new pull request, #22415: [Fix](executor)Fix incorrect mem_limit return value type
wangbo opened a new pull request, #22415: URL: https://github.com/apache/doris/pull/22415 ## Proposed changes Fix a typo for task group's mem limit, this may cause incorrect calculation for memory gc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman opened a new pull request, #22412: [improvement](file-scan) reduce the min size of file split
morningman opened a new pull request, #22412: URL: https://github.com/apache/doris/pull/22412 ## Proposed changes Reduce from 128MB to 8MB. So that user can set `file_split_size` more flexible. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ixzc opened a new pull request, #22414: [typo][doc]Modify some sql syntax description errors.
ixzc opened a new pull request, #22414: URL: https://github.com/apache/doris/pull/22414 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] xiaokang commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
xiaokang commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1278966598
##
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##
@@ -3120,6 +3120,40 @@ public static void getDdlStmt(DdlStmt ddlStmt, String
dbName, TableIf table, Lis
sb.append(olapTable.skipWriteIndexOnLoad()).append("\"");
}
+// compaction policy
+if (olapTable.getCompactionPolicy() != null &&
!olapTable.getCompactionPolicy().equals("")
+&&
!olapTable.getCompactionPolicy().equals(PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY))
{
+
sb.append(",\n\"").append(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY).append("\"
= \"");
+sb.append(olapTable.getCompactionPolicy()).append("\"");
+}
+
+// time series compaction goal size
+if (olapTable.getTimeSeriesCompactionGoalSizeMbytes() != null
+&&
olapTable.getTimeSeriesCompactionGoalSizeMbytes()
+!=
PropertyAnalyzer.TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES_DEFAULT_VALUE) {
Review Comment:
show default value as normal
##
gensrc/thrift/AgentService.thrift:
##
@@ -142,6 +142,10 @@ struct TCreateTabletReq {
19: optional bool enable_unique_key_merge_on_write = false
20: optional i64 storage_policy_id
21: optional TBinlogConfig binlog_config
+22: optional string compaction_policy = "size_based"
+23: optional i64 time_series_compaction_goal_size_mbytes = 512
Review Comment:
default 1024
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##
@@ -142,6 +182,65 @@ public void analyze(Analyzer analyzer) throws
AnalysisException {
||
properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
||
properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in
SchemaChangeHandler.updateBinlogConfig
+} else if
(properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+String compactionPolicy =
properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+if (compactionPolicy != null
+&&
!compactionPolicy.equals(PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY)
+&&
!compactionPolicy.equals(PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY)) {
+throw new AnalysisException(
+"Table compaction policy only support for " +
PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY
++ " or " +
PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY);
+}
+this.needTableStable = false;
+setCompactionPolicy(compactionPolicy);
+} else if
(properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES))
{
+long goalSizeMbytes;
+String goalSizeMbytesStr = properties
+
.get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+try {
+goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
+if (goalSizeMbytes < 0) {
+throw new AnalysisException("Invalid
time_series_compaction_goal_size_mbytes format: "
++ goalSizeMbytesStr);
+}
+} catch (NumberFormatException e) {
+throw new AnalysisException("Invalid
time_series_compaction_goal_size_mbytes format: "
++ goalSizeMbytesStr);
+}
+this.needTableStable = false;
+setTimeSeriesCompactionGoalSizeMbytes(goalSizeMbytes);
+} else if
(properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD))
{
+long fileCountThreshold;
+String fileCountThresholdStr = properties
+
.get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD);
+try {
+fileCountThreshold = Long.parseLong(fileCountThresholdStr);
+if (fileCountThreshold < 0) {
Review Comment:
10
##
be/src/olap/tablet_manager.cpp:
##
@@ -729,7 +731,10 @@ TabletSharedPtr
TabletManager::find_best_tablet_to_compaction(
continue;
}
}
-
+auto cumulative_compaction_policy =
+tablet_ptr->tablet_meta()->compaction_policy() ==
CUMULATIVE_TIME_SERIES_POLICY
Review Comment:
at(tablet_ptr->tablet_meta()->compaction_policy())
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##
@@ -142,6 +182,65 @@ public void an
[GitHub] [doris] ixzc closed pull request #22414: [typo][doc]Modify some sql syntax description errors.
ixzc closed pull request #22414: [typo][doc]Modify some sql syntax description errors. URL: https://github.com/apache/doris/pull/22414 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] ixzc closed pull request #22413: [typo][doc]Modify some sql syntax description errors.
ixzc closed pull request #22413: [typo][doc]Modify some sql syntax description errors. URL: https://github.com/apache/doris/pull/22413 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] wangbo commented on pull request #22415: [Fix](executor)Fix incorrect mem_limit return value type
wangbo commented on PR #22415: URL: https://github.com/apache/doris/pull/22415#issuecomment-1657966744 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1279003905
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##
@@ -142,6 +182,65 @@ public void analyze(Analyzer analyzer) throws
AnalysisException {
||
properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
||
properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in
SchemaChangeHandler.updateBinlogConfig
+} else if
(properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+String compactionPolicy =
properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+if (compactionPolicy != null
+&&
!compactionPolicy.equals(PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY)
+&&
!compactionPolicy.equals(PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY)) {
+throw new AnalysisException(
+"Table compaction policy only support for " +
PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY
++ " or " +
PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY);
+}
+this.needTableStable = false;
+setCompactionPolicy(compactionPolicy);
+} else if
(properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES))
{
+long goalSizeMbytes;
+String goalSizeMbytesStr = properties
+
.get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+try {
+goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
+if (goalSizeMbytes < 0) {
Review Comment:
done
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##
@@ -142,6 +182,65 @@ public void analyze(Analyzer analyzer) throws
AnalysisException {
||
properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
||
properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in
SchemaChangeHandler.updateBinlogConfig
+} else if
(properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+String compactionPolicy =
properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+if (compactionPolicy != null
+&&
!compactionPolicy.equals(PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY)
+&&
!compactionPolicy.equals(PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY)) {
+throw new AnalysisException(
+"Table compaction policy only support for " +
PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY
++ " or " +
PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY);
+}
+this.needTableStable = false;
+setCompactionPolicy(compactionPolicy);
+} else if
(properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES))
{
+long goalSizeMbytes;
+String goalSizeMbytesStr = properties
+
.get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+try {
+goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
+if (goalSizeMbytes < 0) {
+throw new AnalysisException("Invalid
time_series_compaction_goal_size_mbytes format: "
++ goalSizeMbytesStr);
+}
+} catch (NumberFormatException e) {
+throw new AnalysisException("Invalid
time_series_compaction_goal_size_mbytes format: "
++ goalSizeMbytesStr);
+}
+this.needTableStable = false;
+setTimeSeriesCompactionGoalSizeMbytes(goalSizeMbytes);
+} else if
(properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD))
{
+long fileCountThreshold;
+String fileCountThresholdStr = properties
+
.get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD);
+try {
+fileCountThreshold = Long.parseLong(fileCountThresholdStr);
+if (fileCountThreshold < 0) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---
[GitHub] [doris] hello-stephen commented on pull request #22406: [opt](conf) Modify brpc work pool conf default value
hello-stephen commented on PR #22406: URL: https://github.com/apache/doris/pull/22406#issuecomment-1657968088 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.22 seconds stream load tsv: 511 seconds loaded 74807831229 Bytes, about 139 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 29.2 seconds inserted 1000 Rows, about 342K ops/s storage size: 17162377124 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1279004649
##
fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java:
##
@@ -142,6 +182,65 @@ public void analyze(Analyzer analyzer) throws
AnalysisException {
||
properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_BYTES)
||
properties.containsKey(PropertyAnalyzer.PROPERTIES_BINLOG_MAX_HISTORY_NUMS)) {
// do nothing, will be alter in
SchemaChangeHandler.updateBinlogConfig
+} else if
(properties.containsKey(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY)) {
+String compactionPolicy =
properties.getOrDefault(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY, "");
+if (compactionPolicy != null
+&&
!compactionPolicy.equals(PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY)
+&&
!compactionPolicy.equals(PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY)) {
+throw new AnalysisException(
+"Table compaction policy only support for " +
PropertyAnalyzer.TIME_SERIES_COMPACTION_POLICY
++ " or " +
PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY);
+}
+this.needTableStable = false;
+setCompactionPolicy(compactionPolicy);
+} else if
(properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES))
{
+long goalSizeMbytes;
+String goalSizeMbytesStr = properties
+
.get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES);
+try {
+goalSizeMbytes = Long.parseLong(goalSizeMbytesStr);
+if (goalSizeMbytes < 0) {
+throw new AnalysisException("Invalid
time_series_compaction_goal_size_mbytes format: "
++ goalSizeMbytesStr);
+}
+} catch (NumberFormatException e) {
+throw new AnalysisException("Invalid
time_series_compaction_goal_size_mbytes format: "
++ goalSizeMbytesStr);
+}
+this.needTableStable = false;
+setTimeSeriesCompactionGoalSizeMbytes(goalSizeMbytes);
+} else if
(properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD))
{
+long fileCountThreshold;
+String fileCountThresholdStr = properties
+
.get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_FILE_COUNT_THRESHOLD);
+try {
+fileCountThreshold = Long.parseLong(fileCountThresholdStr);
+if (fileCountThreshold < 0) {
+throw new AnalysisException("Invalid
time_series_compaction_file_count_threshold format: "
+
+ fileCountThresholdStr);
+}
+} catch (NumberFormatException e) {
+throw new AnalysisException("Invalid
time_series_compaction_file_count_threshold format: "
+
+ fileCountThresholdStr);
+}
+this.needTableStable = false;
+setTimeSeriesCompactionFileCountThreshold(fileCountThreshold);
+} else if
(properties.containsKey(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_TIME_THRESHOLD_SECONDS))
{
+long timeThresholdSeconds;
+String timeThresholdSecondsStr = properties
+
.get(PropertyAnalyzer.PROPERTIES_TIME_SERIES_COMPACTION_TIME_THRESHOLD_SECONDS);
+try {
+timeThresholdSeconds = Long.parseLong(timeThresholdSecondsStr);
+if (timeThresholdSeconds < 0) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #22382: [feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load
TangSiyang2001 commented on PR #22382: URL: https://github.com/apache/doris/pull/22382#issuecomment-1657969005 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1279005566
##
gensrc/thrift/AgentService.thrift:
##
@@ -142,6 +142,10 @@ struct TCreateTabletReq {
19: optional bool enable_unique_key_merge_on_write = false
20: optional i64 storage_policy_id
21: optional TBinlogConfig binlog_config
+22: optional string compaction_policy = "size_based"
+23: optional i64 time_series_compaction_goal_size_mbytes = 512
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22415: [Fix](executor)Fix incorrect mem_limit return value type
github-actions[bot] commented on PR #22415: URL: https://github.com/apache/doris/pull/22415#issuecomment-1657969529 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22415: [Fix](executor)Fix incorrect mem_limit return value type
github-actions[bot] commented on PR #22415: URL: https://github.com/apache/doris/pull/22415#issuecomment-1657969642 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] yiguolei commented on pull request #22409: [feature](ui) add profile download button in the list page
yiguolei commented on PR #22409: URL: https://github.com/apache/doris/pull/22409#issuecomment-1657971888 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22409: [feature](ui) add profile download button in the list page
github-actions[bot] commented on PR #22409: URL: https://github.com/apache/doris/pull/22409#issuecomment-1657974322 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22415: [Fix](executor)Fix incorrect mem_limit return value type
github-actions[bot] commented on PR #22415: URL: https://github.com/apache/doris/pull/22415#issuecomment-1657974367 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22409: [feature](ui) add profile download button in the list page
github-actions[bot] commented on PR #22409: URL: https://github.com/apache/doris/pull/22409#issuecomment-1657974401 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] bobhan1 opened a new pull request, #22416: [fix](window_funnel_function) fix upgrade compatibility due to the added field in `WindowFunnelState`
bobhan1 opened a new pull request, #22416: URL: https://github.com/apache/doris/pull/22416 ## Proposed changes the added variable `window_funnel_mode` in `WindowFunnelState` will cause the serialization result of new BE different from old version BE, which may lead to coredump when new version BE and old version BE deserialize from each others serialization result. So this PR add a config variable to control if we use the new window funnel function. This config should be turned off during the process of upgrade. After all the BE have upgrade to the new version the config can be turned on to use the new version of window_funnel_function ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1279015472
##
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##
@@ -3120,6 +3120,40 @@ public static void getDdlStmt(DdlStmt ddlStmt, String
dbName, TableIf table, Lis
sb.append(olapTable.skipWriteIndexOnLoad()).append("\"");
}
+// compaction policy
+if (olapTable.getCompactionPolicy() != null &&
!olapTable.getCompactionPolicy().equals("")
+&&
!olapTable.getCompactionPolicy().equals(PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY))
{
+
sb.append(",\n\"").append(PropertyAnalyzer.PROPERTIES_COMPACTION_POLICY).append("\"
= \"");
+sb.append(olapTable.getCompactionPolicy()).append("\"");
+}
+
+// time series compaction goal size
+if (olapTable.getTimeSeriesCompactionGoalSizeMbytes() != null
+&&
olapTable.getTimeSeriesCompactionGoalSizeMbytes()
+!=
PropertyAnalyzer.TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES_DEFAULT_VALUE) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morningman commented on pull request #22371: [refactor](reader) move reader from vec/exec/scan to vec/exec/format
morningman commented on PR #22371: URL: https://github.com/apache/doris/pull/22371#issuecomment-1657977635 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] TangSiyang2001 commented on pull request #22382: [feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load
TangSiyang2001 commented on PR #22382: URL: https://github.com/apache/doris/pull/22382#issuecomment-1657978160 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
csun5285 commented on code in PR #22159: URL: https://github.com/apache/doris/pull/22159#discussion_r1279017706 ## be/src/olap/tablet_manager.cpp: ## @@ -729,7 +731,10 @@ TabletSharedPtr TabletManager::find_best_tablet_to_compaction( continue; } } - +auto cumulative_compaction_policy = +tablet_ptr->tablet_meta()->compaction_policy() == CUMULATIVE_TIME_SERIES_POLICY Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1279021089
##
be/src/olap/olap_server.cpp:
##
@@ -976,8 +981,18 @@ Status
StorageEngine::_submit_compaction_task(TabletSharedPtr tablet,
Status StorageEngine::submit_compaction_task(TabletSharedPtr tablet,
CompactionType compaction_type,
bool force) {
_update_cumulative_compaction_policy();
-if (tablet->get_cumulative_compaction_policy() == nullptr) {
-
tablet->set_cumulative_compaction_policy(_cumulative_compaction_policy);
+// alter table tableName set ("compaction_policy"="time_series")
+// if atler table's compaction policy, we need to modify tablet
compaction policy shared ptr
+if (tablet->get_cumulative_compaction_policy() == nullptr ||
+tablet->get_cumulative_compaction_policy()->name() !=
+tablet->tablet_meta()->compaction_policy()) {
+if (tablet->tablet_meta()->compaction_policy() ==
CUMULATIVE_TIME_SERIES_POLICY) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] bobhan1 commented on pull request #22416: [fix](window_funnel_function) fix upgrade compatibility due to the added field in `WindowFunnelState`
bobhan1 commented on PR #22416: URL: https://github.com/apache/doris/pull/22416#issuecomment-1657984243 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22412: [improvement](file-scan) reduce the min size of file split
github-actions[bot] commented on PR #22412: URL: https://github.com/apache/doris/pull/22412#issuecomment-1657985559 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22412: [improvement](file-scan) reduce the min size of file split
github-actions[bot] commented on PR #22412: URL: https://github.com/apache/doris/pull/22412#issuecomment-1657985656 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] csun5285 commented on a diff in pull request #22159: [improvement](compaction) Configuring compaction policy and options in the properties of a table
csun5285 commented on code in PR #22159:
URL: https://github.com/apache/doris/pull/22159#discussion_r1279023738
##
regression-test/suites/compaction/test_table_level_compaction_policy.groovy:
##
@@ -0,0 +1,131 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.codehaus.groovy.runtime.IOGroovyMethods
+
+suite("test_table_level_compaction_policy") {
+def tableName = "test_table_level_compaction_policy"
+sql """ DROP TABLE IF EXISTS ${tableName} """
+
+sql """
+CREATE TABLE ${tableName} (
+`c_custkey` int(11) NOT NULL COMMENT "",
+`c_name` varchar(26) NOT NULL COMMENT "",
+`c_address` varchar(41) NOT NULL COMMENT "",
+`c_city` varchar(11) NOT NULL COMMENT ""
+)
+DUPLICATE KEY (`c_custkey`)
+DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1
+PROPERTIES (
+"replication_num" = "1",
+"compaction_policy" = "time_series",
+"time_series_compaction_goal_size_mbytes" = "1024",
+"time_series_compaction_file_count_threshold" = "5000",
+"time_series_compaction_time_threshold_seconds" = "86400"
+ );
+"""
+result = sql """show create table ${tableName}"""
+logger.info("${result}")
+assertTrue(result.toString().containsIgnoreCase('"compaction_policy" =
"time_series"'))
+
assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_goal_size_mbytes"
= "1024"'))
+
assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_file_count_threshold"
= "5000"'))
+
assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_time_threshold_seconds"
= "86400"'))
+
+sql """
+alter table ${tableName} set
("time_series_compaction_goal_size_mbytes" = "2048")
+"""
+
+result = sql """show create table ${tableName}"""
+logger.info("${result}")
+
assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_goal_size_mbytes"
= "2048"'))
+
+sql """
+alter table ${tableName} set
("time_series_compaction_file_count_threshold" = "6000")
+"""
+
+result = sql """show create table ${tableName}"""
+logger.info("${result}")
+
assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_file_count_threshold"
= "6000"'))
+
+ sql """
+alter table ${tableName} set
("time_series_compaction_time_threshold_seconds" = "3000")
+"""
+
+result = sql """show create table ${tableName}"""
+logger.info("${result}")
+
assertTrue(result.toString().containsIgnoreCase('"time_series_compaction_time_threshold_seconds"
= "3000"'))
+
+sql """ DROP TABLE IF EXISTS ${tableName} """
+
+test {
+sql """
+CREATE TABLE ${tableName} (
+`c_custkey` int(11) NOT NULL COMMENT "",
+`c_name` varchar(26) NOT NULL COMMENT "",
+`c_address` varchar(41) NOT NULL COMMENT "",
+`c_city` varchar(11) NOT NULL COMMENT ""
+)
+DUPLICATE KEY (`c_custkey`)
+DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1
+PROPERTIES (
+"replication_num" = "1",
+"compaction_policy" = "ok"
+ );
+"""
+exception "compaction_policy must be time_series or size_based"
+}
+
+test {
+sql """
+CREATE TABLE ${tableName} (
+`c_custkey` int(11) NOT NULL COMMENT "",
+`c_name` varchar(26) NOT NULL COMMENT "",
+`c_address` varchar(41) NOT NULL COMMENT "",
+`c_city` varchar(11) NOT NULL COMMENT ""
+)
+DUPLICATE KEY (`c_custkey`)
+DISTRIBUTED BY HASH(`c_custkey`) BUCKETS 1
+PROPERTIES (
+"replication_num" = "1",
+"time_series_compaction_goal_size_mbytes" = "1024"
+ );
+"""
+exception "only time series compaction policy support for time series
config"
+}
+
+t
[GitHub] [doris] starocean999 opened a new pull request, #22417: [fix](planner)fix bug of push conjuncts through second phase agg
starocean999 opened a new pull request, #22417: URL: https://github.com/apache/doris/pull/22417 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #21916: [Feature](Job)Support scheduler job
hello-stephen commented on PR #21916: URL: https://github.com/apache/doris/pull/21916#issuecomment-1657987291 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.85 seconds stream load tsv: 508 seconds loaded 74807831229 Bytes, about 140 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s insert into select: 29.3 seconds inserted 1000 Rows, about 341K ops/s storage size: 17162248240 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #22416: [fix](window_funnel_function) fix upgrade compatibility due to the added field in `WindowFunnelState`
github-actions[bot] commented on PR #22416: URL: https://github.com/apache/doris/pull/22416#issuecomment-1657988029 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #22411: [fix](string-column) fix unescape result length error
hello-stephen commented on PR #22411: URL: https://github.com/apache/doris/pull/22411#issuecomment-1657988953 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 48.78 seconds stream load tsv: 528 seconds loaded 74807831229 Bytes, about 135 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17162127243 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] Yukang-Lian commented on pull request #22382: [feature](load) refactor CSV reading process during scanning, and support enclose and escape for stream load
Yukang-Lian commented on PR #22382: URL: https://github.com/apache/doris/pull/22382#issuecomment-1657989310 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 commented on pull request #22417: [fix](planner)fix bug of push conjuncts through second phase agg
starocean999 commented on PR #22417: URL: https://github.com/apache/doris/pull/22417#issuecomment-1657990197 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] starocean999 commented on pull request #22417: [fix](planner)fix bug of push conjuncts through second phase agg
starocean999 commented on PR #22417: URL: https://github.com/apache/doris/pull/22417#issuecomment-1657990445 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[GitHub] [doris] morrySnow merged pull request #22196: [Fix](Planner) fix parse error of view with group_concat order by
morrySnow merged PR #22196: URL: https://github.com/apache/doris/pull/22196 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [Fix](Planner) fix parse error of view with group_concat order by (#22196)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 3a1d678ca9 [Fix](Planner) fix parse error of view with group_concat
order by (#22196)
3a1d678ca9 is described below
commit 3a1d678ca97f6d7e69a89fbe05c6f701515bb67c
Author: LiBinfeng <46676950+libinfeng...@users.noreply.github.com>
AuthorDate: Mon Jul 31 17:20:23 2023 +0800
[Fix](Planner) fix parse error of view with group_concat order by (#22196)
Problem:
When create view with projection group_concat(xxx, xxx order by
orderkey). It will failed during second parse of inline view
For example:
it works when doing
"SELECT id, group_concat(`name`, "," ORDER BY id) AS test_group_column
FROM test GROUP BY id"
but when create view it does not work
"create view test_view as SELECT id, group_concat(`name`, "," ORDER BY
id) AS test_group_column FROM test GROUP BY id"
Reason:
when creating view, we will doing parse again of view.toSql() to check
whether it has some syntax error. And when doing toSql() to group_concat with
order by, it add seperate ', ' between second parameter and order by. So when
parsing again, it
would failed because it is different semantic with original statement.
group_concat(`name`, "," ORDER BY id) ==> group_concat(`name`, "," ,
ORDER BY id)
Solved:
Change toSql of group_concat and add order by statement analyze() of
group_concat in Planner cause it would work if we get order by from view
statement and do not analyze and binding slot reference to it
---
.../java/org/apache/doris/analysis/FunctionCallExpr.java | 12 +++-
.../data/nereids_p0/group_concat/test_group_concat.out | 4
.../data/query_p0/group_concat/test_group_concat.out | 5 -
.../suites/nereids_p0/group_concat/test_group_concat.groovy | 6 ++
.../suites/query_p0/group_concat/test_group_concat.groovy| 7 ++-
5 files changed, 31 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
index e28587f84f..44d196e18e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/FunctionCallExpr.java
@@ -547,7 +547,12 @@ public class FunctionCallExpr extends Expr {
for (int i = 0; i < len; ++i) {
if (i != 0) {
-sb.append(", ");
+if (fnName.getFunction().equalsIgnoreCase("group_concat")
+&& orderByElements.size() > 0 && i == len -
orderByElements.size()) {
+sb.append(" ");
+} else {
+sb.append(", ");
+}
}
if (ConnectContext.get() != null &&
ConnectContext.get().getState().isQuery() && i == 1
&& (fnName.getFunction().equalsIgnoreCase("aes_decrypt")
@@ -1785,6 +1790,11 @@ public class FunctionCallExpr extends Expr {
}
// rewrite return type if is nested type function
analyzeNestedFunction();
+for (OrderByElement o : orderByElements) {
+if (!o.getExpr().isAnalyzed) {
+o.getExpr().analyzeImpl(analyzer);
+}
+}
}
// if return type is nested type, need to be determined the sub-element
type
diff --git a/regression-test/data/nereids_p0/group_concat/test_group_concat.out
b/regression-test/data/nereids_p0/group_concat/test_group_concat.out
index 16467f6be7..2d97d50122 100644
--- a/regression-test/data/nereids_p0/group_concat/test_group_concat.out
+++ b/regression-test/data/nereids_p0/group_concat/test_group_concat.out
@@ -59,3 +59,7 @@ false
1 3,21,2,11,1
2 23,222,22,211,21
+-- !select_group_concat_order_by_desc4 --
+1 3,21,2,11,1
+2 23,222,22,211,21
+
diff --git a/regression-test/data/query_p0/group_concat/test_group_concat.out
b/regression-test/data/query_p0/group_concat/test_group_concat.out
index 7153ce1be3..d01900ef88 100644
--- a/regression-test/data/query_p0/group_concat/test_group_concat.out
+++ b/regression-test/data/query_p0/group_concat/test_group_concat.out
@@ -63,6 +63,9 @@ false
1 3,21,2,11,1
2 23,222,22,211,21
--- !select_group_concat_order_by --
+-- !select_group_concat_order_by1 --
+1,11,2,21,21,211,22,222,23,3 3,23,222,22,211,21,21,2,11,1
+
+-- !select_group_concat_order_by2 --
1,11,2,21,21,211,22,222,23,3 3,23,222,22,211,21,21,2,11,1
diff --git
a/regression-test/suites/nereids_p0/group_concat/test_group_concat.groovy
b/regression-test/suites/nereids_p0/group_concat/test_group_concat.groovy
index b9896f5cf8..a570ac3da1 100644
--- a/regression-test/suites/nere
