Re: [PR] [performance](variant) support topn 2phase read for variant column [doris]
doris-robot commented on PR #28318: URL: https://github.com/apache/doris/pull/28318#issuecomment-1857441041 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.51 seconds stream load tsv: 606 seconds loaded 74807831229 Bytes, about 117 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17222438823 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [ci](perf) 1. add perf check of tpcds, 2. adjust clickbench and tpch check [doris]
github-actions[bot] commented on PR #28431:
URL: https://github.com/apache/doris/pull/28431#issuecomment-1857442084
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/7219448671";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In regression-test/pipeline/performance/clean.sh line 36:
source ../../common/doris-utils.sh
^-^ SC1091 (info): Not following:
../../common/doris-utils.sh: openBinaryFile: does not exist (No such file or
directory)
In regression-test/pipeline/performance/run-clickbench.sh line 81:
bash
"${teamcity_build_checkoutDir}"/tools/clickbench-tools/bin/create-clickbench-tables.sh
-s "${SF}"
^---^ SC2154 (warning): SF is referenced but not
assigned.
For more information:
https://www.shellcheck.net/wiki/SC2154 -- SF is referenced but not
assigned.
https://www.shellcheck.net/wiki/SC1091 -- Not following:
../../common/doris...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' returned error 1 finding the following formatting issues:
--
--- regression-test/pipeline/performance/run-clickbench.sh.orig
+++ regression-test/pipeline/performance/run-clickbench.sh
@@ -69,14 +69,14 @@
if ! restart_doris; then echo "ERROR: Restart doris failed" && exit 1;
fi
echo " 2. check if need to load data"
-CLICKBENCH_DATA_DIR="/data/clickbench"
# no / at the end
+CLICKBENCH_DATA_DIR="/data/clickbench"
# no / at the end
CLICKBENCH_DATA_DIR_LINK="${teamcity_build_checkoutDir}"/tools/clickbench-tools/data
# no / at the end
db_name="clickbench"
if ! check_clickbench_table_rows "${db_name}"; then
echo "INFO: need to load clickbench data"
# prepare data
mkdir -p "${CLICKBENCH_DATA_DIR}"
-
+
# create table and load data
bash
"${teamcity_build_checkoutDir}"/tools/clickbench-tools/bin/create-clickbench-tables.sh
-s "${SF}"
rm -rf "${CLICKBENCH_DATA_DIR_LINK}"
--
You can reformat the above files to meet shfmt's requirements by typing:
shfmt -w filename
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refact](fe) Split `SchemaChangeHandler.process` into several function [doris]
doris-robot commented on PR #28366: URL: https://github.com/apache/doris/pull/28366#issuecomment-1857442974 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.25 seconds stream load tsv: 585 seconds loaded 74807831229 Bytes, about 121 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 29.1 seconds inserted 1000 Rows, about 343K ops/s storage size: 17218291372 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [DNM](Nereids) test disable fallback [doris]
doris-robot commented on PR #25344: URL: https://github.com/apache/doris/pull/25344#issuecomment-1857444875 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 43.85 seconds stream load tsv: 588 seconds loaded 74807831229 Bytes, about 121 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17222724183 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block [doris]
caiconghui commented on PR #27943: URL: https://github.com/apache/doris/pull/27943#issuecomment-1857443729 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](publish) Publish task not wait when be's publish task a… [doris]
deardeng commented on code in PR #28079:
URL: https://github.com/apache/doris/pull/28079#discussion_r1427673698
##
fe/fe-core/src/main/java/org/apache/doris/transaction/PublishVersionDaemon.java:
##
@@ -132,27 +133,45 @@ private void publishVersion() {
Map tableIdToTotalDeltaNumRows = Maps.newHashMap();
// try to finish the transaction, if failed just retry in next loop
for (TransactionState transactionState : readyTransactionStates) {
-Stream publishVersionTaskStream =
transactionState
-.getPublishVersionTasks()
-.values()
-.stream()
-.peek(task -> {
-if (task.isFinished() &&
CollectionUtils.isEmpty(task.getErrorTablets())) {
-Map tableIdToDeltaNumRows =
-task.getTableIdToDeltaNumRows();
-tableIdToDeltaNumRows.forEach((tableId, numRows)
-> {
-tableIdToTotalDeltaNumRows
-.computeIfPresent(tableId, (id,
orgNumRows) -> orgNumRows + numRows);
-
tableIdToTotalDeltaNumRows.putIfAbsent(tableId, numRows);
-});
-}
-});
-boolean hasBackendAliveAndUnfinishedTask = publishVersionTaskStream
-.anyMatch(task -> !task.isFinished() &&
infoService.checkBackendAlive(task.getBackendId()));
+AtomicReference finishNum = new AtomicReference<>(0L);
+AtomicBoolean hasBackendAliveAndUnfinishedTask = new
AtomicBoolean(false);
+Set notFinishTaskBe = Sets.newHashSet();
+transactionState.getPublishVersionTasks().forEach((beId, task) -> {
+if (task.isFinished()) {
+finishNum.set(finishNum.get() + 1);
+if (CollectionUtils.isEmpty(task.getErrorTablets())) {
+Map tableIdToDeltaNumRows =
task.getTableIdToDeltaNumRows();
+tableIdToDeltaNumRows.forEach((tableId, numRows) -> {
+tableIdToTotalDeltaNumRows
+.computeIfPresent(tableId, (id, orgNumRows) ->
orgNumRows + numRows);
+tableIdToTotalDeltaNumRows.putIfAbsent(tableId,
numRows);
+});
+}
+} else {
+if (infoService.checkBackendAlive(task.getBackendId())) {
+hasBackendAliveAndUnfinishedTask.set(true);
+}
+notFinishTaskBe.add(beId);
+}
+});
+
transactionState.setTableIdToTotalNumDeltaRows(tableIdToTotalDeltaNumRows);
+LOG.debug("notFinishTaskBe {}, trans {}", notFinishTaskBe,
transactionState);
+boolean isPublishSlow = false;
+if (finishNum.get() >
transactionState.getPublishVersionTasks().keySet().size() / 2
+&& notFinishTaskBe.stream()
+.allMatch(beId ->
infoService.getBackend(beId).getPublishTaskLastTimeAccumulated())) {
+LOG.debug(" finishNum {}, txn publish tasks {},
notFinishTaskBe {}",
+finishNum,
transactionState.getPublishVersionTasks().keySet(), notFinishTaskBe);
+isPublishSlow = true;
+}
-boolean shouldFinishTxn = !hasBackendAliveAndUnfinishedTask ||
transactionState.isPublishTimeout()
+boolean shouldFinishTxn = !hasBackendAliveAndUnfinishedTask.get()
|| transactionState.isPublishTimeout()
+|| isPublishSlow
||
DebugPointUtil.isEnable("PublishVersionDaemon.not_wait_unfinished_tasks");
+LOG.debug("hasBackendAliveAndUnfinishedTask {}, txn
isPublishTimeout {}, DebugPoint enable {}, slow {}",
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) explain should fallback too if Nereids is not enable [doris]
doris-robot commented on PR #28475: URL: https://github.com/apache/doris/pull/28475#issuecomment-1857453817 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.36 seconds stream load tsv: 581 seconds loaded 74807831229 Bytes, about 122 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 27.8 seconds inserted 1000 Rows, about 359K ops/s storage size: 17222678322 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block [doris]
github-actions[bot] commented on PR #27943: URL: https://github.com/apache/doris/pull/27943#issuecomment-1857454660 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [nereids] found rows support fe side [doris]
xzj7019 commented on PR #28372: URL: https://github.com/apache/doris/pull/28372#issuecomment-1857455471 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](hint) query fail print tablet detail info [doris]
doris-robot commented on PR #28476: URL: https://github.com/apache/doris/pull/28476#issuecomment-1857455561 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 43.99 seconds stream load tsv: 590 seconds loaded 74807831229 Bytes, about 120 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17222838080 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](full compaction) Full compaction should hold meta lock when modifying tablet's meta data [doris]
bobhan1 commented on PR #28449: URL: https://github.com/apache/doris/pull/28449#issuecomment-1857456250 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](session variables) Make default value of max_execution_time same to query_timeout [doris]
doris-robot commented on PR #28474: URL: https://github.com/apache/doris/pull/28474#issuecomment-1857457024 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.59 seconds stream load tsv: 585 seconds loaded 74807831229 Bytes, about 121 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17217872964 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.0] Pick "[fix](partial update) Fix missing rowsets during doing alignment when flushing memtable due to compaction #28062" [doris]
doris-robot commented on PR #28468: URL: https://github.com/apache/doris/pull/28468#issuecomment-1857458954 TeamCity be ut coverage result: Function Coverage: 37.81% (7988/21129) Line Coverage: 29.48% (64826/219913) Region Coverage: 28.95% (33356/115238) Branch Coverage: 24.84% (17122/68938) Coverage Report: http://coverage.selectdb-in.cc/coverage/cce3823cea1961379ddc7ca91964793abcb1983c_cce3823cea1961379ddc7ca91964793abcb1983c/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](Load)(step2)support nereids load job schedule [doris]
wsjz commented on code in PR #26356:
URL: https://github.com/apache/doris/pull/26356#discussion_r1427678062
##
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/InsertTask.java:
##
@@ -74,30 +72,70 @@ public class InsertTask extends AbstractTask {
}
private String labelName;
Review Comment:
now use for getTvfInfo()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Pick-2.0](vcompound pred) Corrected evaluation for compound predicates with constant columns [doris]
doris-robot commented on PR #28477: URL: https://github.com/apache/doris/pull/28477#issuecomment-1857459190 TeamCity be ut coverage result: Function Coverage: 37.88% (8002/21126) Line Coverage: 29.54% (64938/219862) Region Coverage: 29.01% (33425/115215) Branch Coverage: 24.90% (17160/68926) Coverage Report: http://coverage.selectdb-in.cc/coverage/75dddb5158045acb627754f7f5660c1d95dda91a_75dddb5158045acb627754f7f5660c1d95dda91a/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](Load)(step2)support nereids load job schedule [doris]
wsjz commented on code in PR #26356:
URL: https://github.com/apache/doris/pull/26356#discussion_r1427678969
##
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/InsertTask.java:
##
@@ -74,30 +72,70 @@ public class InsertTask extends AbstractTask {
}
private String labelName;
-
private InsertIntoTableCommand command;
-
private StmtExecutor stmtExecutor;
-
private ConnectContext ctx;
-
private String sql;
-
private String currentDb;
-
private UserIdentity userIdentity;
-
+private LoadStatistic loadStatistic;
private AtomicBoolean isCanceled = new AtomicBoolean(false);
-
private AtomicBoolean isFinished = new AtomicBoolean(false);
-
private static final String LABEL_SPLITTER = "_";
+private FailMsg failMsg;
+@Getter
+private String trackingUrl;
@Getter
@Setter
-private LoadJob loadJob;
+private InsertJob jobInfo;
+private TaskType taskType = TaskType.PENDING;
+private MergeType mergeType = MergeType.APPEND;
+
+/**
+ * task merge type
+ */
+enum MergeType {
+MERGE,
+APPEND,
+DELETE
+}
+
+/**
+ * task type
+ */
+enum TaskType {
+UNKNOWN, // this is only for ISSUE #2354
+PENDING,
+LOADING,
+FINISHED,
+FAILED,
+CANCELLED
+}
+
+public InsertTask(InsertIntoTableCommand insertInto,
+ ConnectContext ctx, StmtExecutor executor, LoadStatistic
statistic) {
+this(null, insertInto, ctx, executor, statistic);
+}
+
+public InsertTask(String labelName, String currentDb, String sql,
UserIdentity userIdentity) {
+this.labelName = labelName;
+this.sql = sql;
+this.currentDb = currentDb;
+this.userIdentity = userIdentity;
+setTaskId(Env.getCurrentEnv().getNextId());
+}
+public InsertTask(String labelName, InsertIntoTableCommand insertInto,
+ ConnectContext ctx, StmtExecutor executor,
LoadStatistic statistic) {
+this.labelName = labelName;
Review Comment:
the first constructor don't need loadStatistic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[I] [Feature] 窗口分析函数支持窗口按照时间范围 [doris]
xiaolei55 opened a new issue, #28478: URL: https://github.com/apache/doris/issues/28478 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description 窗口分析函数支持按照时间窗口，类似语法 `SELECT agg_func(agg_col) OVER ( [PARTITION BY col1[, col2, ...]] ORDER BY time_col range_definition), ... FROM ...` range_definition从句可以支持按照ROWS（已实现）和RANGE（未实现）， 其中RANGE的使用可以指定时间区间，例如统计当前行前30分钟的数据，就可以使用下面的range_definition来表示 `RANGE BETWEEN INTERVAL '30' MINUTE PRECEDING AND CURRENT ROW` 当然也支持前后时间范围，例如统计当前行前后30分钟的数据 `RANGE BETWEEN INTERVAL '30' MINUTE PRECEDING AND INTERVAL '30' MINUTE FOLLOWING` ### Use case 解决按照时间范围的窗口统计，丰富窗口分析函数的使用场景 ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Planner): parse more Punctuation Date/DateTime [doris]
jackwener merged PR #28432: URL: https://github.com/apache/doris/pull/28432 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](Planner): parse more Punctuation Date/DateTime (#28432)
This is an automated email from the ASF dual-hosted git repository.
jakevin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8986bb6bb4d [fix](Planner): parse more Punctuation Date/DateTime
(#28432)
8986bb6bb4d is described below
commit 8986bb6bb4d600cf1e79672f36f5e5eb40e1424a
Author: jakevin
AuthorDate: Fri Dec 15 16:17:44 2023 +0800
[fix](Planner): parse more Punctuation Date/DateTime (#28432)
parse more Punctuation as separator, like `2021@01@01 00/00/00`;
---
.../trees/expressions/literal/DateLiteral.java | 121 +++--
.../trees/expressions/literal/DateLiteralTest.java | 107 +++---
.../expressions/literal/DateTimeLiteralTest.java | 23 +---
.../data/correctness/test_cast_as_time.out | 2 +-
4 files changed, 185 insertions(+), 68 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
index 19720b68dc3..f2a136ae771 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateLiteral.java
@@ -29,10 +29,13 @@ import org.apache.doris.nereids.util.DateTimeFormatterUtils;
import org.apache.doris.nereids.util.DateUtils;
import org.apache.doris.nereids.util.StandardDateFormat;
+import com.google.common.collect.ImmutableSet;
+
import java.time.LocalDateTime;
import java.time.Year;
import java.time.temporal.ChronoField;
import java.time.temporal.TemporalAccessor;
+import java.util.Set;
/**
* Date literal in Nereids.
@@ -47,6 +50,10 @@ public class DateLiteral extends Literal {
private static final DateLiteral MAX_DATE = new DateLiteral(, 12, 31);
private static final int[] DAYS_IN_MONTH = new int[] {0, 31, 28, 31, 30,
31, 30, 31, 31, 30, 31, 30, 31};
+private static final Set punctuations = ImmutableSet.of('!',
'@', '#', '$', '%', '^', '&', '*', '(', ')',
+'-', '+', '=', '_', '{', '}', '[', ']', '|', '\\', ':', ';', '"',
'\'', '<', '>', ',', '.', '?', '/', '~',
+'`');
+
protected long year;
protected long month;
protected long day;
@@ -133,16 +140,34 @@ public class DateLiteral extends Literal {
return s;
}
+private static boolean isPunctuation(char c) {
+return punctuations.contains(c);
+}
+
+private static void replacePunctuation(String s, StringBuilder sb, char c,
int idx) {
+if (idx >= sb.length()) {
+return;
+}
+if (isPunctuation(sb.charAt(idx))) {
+sb.setCharAt(idx, c);
+} else {
+throw new AnalysisException("date/datetime literal [" + s + "] is
invalid");
+}
+}
+
static String normalize(String s) {
+// merge consecutive space
+s = s.replaceAll(" +", " ");
+
StringBuilder sb = new StringBuilder();
int i = 0;
// handle two digit year
-if (s.charAt(2) != '-' && s.charAt(4) != '-') {
+if (!isPunctuation(s.charAt(2)) && !isPunctuation(s.charAt(4))) {
throw new AnalysisException("date/datetime literal [" + s + "] is
invalid");
}
-if (s.charAt(2) == '-') {
+if (isPunctuation(s.charAt(2))) {
String yy = s.substring(0, 2);
int year = Integer.parseInt(yy);
if (year >= 0 && year <= 69) {
@@ -154,21 +179,12 @@ public class DateLiteral extends Literal {
i = 2;
}
-// normalized leading 0
+// normalize leading 0 for date and time
+// date and time contains 6 number part at most, so we just need
normal 6 number part
+int partNumber = 0;
while (i < s.length()) {
char c = s.charAt(i);
-
-if (c == '.') {
-// skip .microsecond, such as .0001 .01
-sb.append(c); // Append the dot itself
-i += 1; // Skip the dot
-
-// skip the microsecond part
-while (i < s.length() && Character.isDigit(s.charAt(i))) {
-sb.append(s.charAt(i));
-i += 1;
-}
-} else if (Character.isDigit(c)) {
+if (Character.isDigit(c) && partNumber < 6) {
// find consecutive digit
int j = i + 1;
while (j < s.length() && Character.isDigit(s.charAt(j))) {
@@ -180,40 +196,67 @@ public class DateLiteral extends Literal {
sb.append(s.charAt(k));
}
} else if (len == 1) {
-sb.append('0');
-sb.append(c);
+sb.append('0').append(c);
Re: [PR] [fix](Nereids): TransposeSemiJoinAgg can't apply in Scalar Agg [doris]
jackwener merged PR #28434: URL: https://github.com/apache/doris/pull/28434 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](full compaction) Full compaction should hold meta lock when modifying tablet's meta data [doris]
github-actions[bot] commented on PR #28449: URL: https://github.com/apache/doris/pull/28449#issuecomment-1857463843 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (8986bb6bb4d -> 0f93ee87936)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 8986bb6bb4d [fix](Planner): parse more Punctuation Date/DateTime (#28432) add 0f93ee87936 [fix](Nereids): TransposeSemiJoinAgg can't apply in Scalar Agg (#28434) No new revisions were added by this update. Summary of changes: .../doris/nereids/rules/rewrite/TransposeSemiJoinAgg.java| 4 .../nereids/rules/rewrite/TransposeSemiJoinAggTest.java | 12 2 files changed, 16 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](user) Add user property `parallel_fragment_exec_instance_num` [doris]
doris-robot commented on PR #28447: URL: https://github.com/apache/doris/pull/28447#issuecomment-1857464612 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.45 seconds stream load tsv: 586 seconds loaded 74807831229 Bytes, about 121 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17219793120 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Pick-2.0](vcompound pred) Corrected evaluation for compound predicates with constant columns [doris]
doris-robot commented on PR #28477: URL: https://github.com/apache/doris/pull/28477#issuecomment-1857466212 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 51.11 seconds stream load tsv: 603 seconds loaded 74807831229 Bytes, about 118 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.5 seconds inserted 1000 Rows, about 338K ops/s storage size: 17164015687 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [regression](memtable) add case for memtable flush error status handle [doris]
Ma1oneZhang commented on PR #28285: URL: https://github.com/apache/doris/pull/28285#issuecomment-1857469600 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [regression](memtable) add case for memtable flush error status handle [doris]
github-actions[bot] commented on PR #28285: URL: https://github.com/apache/doris/pull/28285#issuecomment-1857472765 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](Load)(step2)support nereids load job schedule [doris]
wsjz commented on code in PR #26356:
URL: https://github.com/apache/doris/pull/26356#discussion_r1427690881
##
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/InsertJob.java:
##
@@ -65,58 +100,171 @@ public class InsertJob extends AbstractJob {
new Column("CreateTime", ScalarType.createStringType()),
new Column("Comment", ScalarType.createStringType()));
+private static final ShowResultSetMetaData TASK_META_DATA =
+ShowResultSetMetaData.builder()
+.addColumn(new Column("TaskId",
ScalarType.createVarchar(20)))
+.addColumn(new Column("Label",
ScalarType.createVarchar(20)))
+.addColumn(new Column("Status",
ScalarType.createVarchar(20)))
+.addColumn(new Column("EtlInfo",
ScalarType.createVarchar(20)))
+.addColumn(new Column("TaskInfo",
ScalarType.createVarchar(20)))
+.addColumn(new Column("ErrorMsg",
ScalarType.createVarchar(20)))
+
+.addColumn(new Column("CreateTimeMs",
ScalarType.createVarchar(20)))
+.addColumn(new Column("FinishTimeMs",
ScalarType.createVarchar(20)))
+.addColumn(new Column("TrackingUrl",
ScalarType.createVarchar(20)))
+.addColumn(new Column("LoadStatistic",
ScalarType.createVarchar(20)))
+.addColumn(new Column("User",
ScalarType.createVarchar(20)))
+.build();
+
public static final ImmutableMap COLUMN_TO_INDEX;
static {
-ImmutableMap.Builder builder = new
ImmutableMap.Builder();
+ImmutableMap.Builder builder = new
ImmutableMap.Builder<>();
for (int i = 0; i < SCHEMA.size(); i++) {
builder.put(SCHEMA.get(i).getName().toLowerCase(), i);
}
COLUMN_TO_INDEX = builder.build();
}
-@SerializedName(value = "lp")
-String labelPrefix;
+@SerializedName("taskIdList")
+ConcurrentLinkedQueue taskIdList;
+@SerializedName("dbId")
+private final long dbId;
+@SerializedName("labelName")
+private String labelName;
+@SerializedName("loadType")
+private InsertJob.LoadType loadType;
+// 0: the job status is pending
+// n/100: n is the number of task which has been finished
+// 99: all tasks have been finished
+// 100: txn status is visible and load has been finished
+@SerializedName("progress")
+private int progress;
+@SerializedName("failMsg")
+private FailMsg failMsg;
+@SerializedName("plans")
+private List plans;
+private LoadStatistic loadStatistic = new LoadStatistic();
+private Set finishedTaskIds = new HashSet<>();
+private Set tableNames;
+private ConcurrentHashMap idToTasks = new
ConcurrentHashMap<>();
+private Map properties;
+private AuthorizationInfo authorizationInfo;
+private final ReentrantReadWriteLock lock = new
ReentrantReadWriteLock(true);
+
+private ConnectContext ctx;
+private StmtExecutor stmtExecutor;
+private List errorTabletInfos = new ArrayList<>();
+private List commitInfos = new ArrayList<>();
+
+// max save task num, do we need to config it?
+private static final int MAX_SAVE_TASK_NUM = 100;
+
+/**
+ * load job type
+ */
+public enum LoadType {
+BULK,
+SPARK,
+LOCAL_FILE,
+UNKNOWN
+
+}
+
+public enum Priority {
+HIGH(0),
+NORMAL(1),
+LOW(2);
-InsertIntoTableCommand command;
+Priority(int value) {
+this.value = value;
+}
-StmtExecutor stmtExecutor;
+private final int value;
-ConnectContext ctx;
+public int getValue() {
+return value;
+}
+}
-@SerializedName("tis")
-ConcurrentLinkedQueue taskIdList;
+public InsertJob(Long jobId, String jobName,
+ JobStatus jobStatus,
+ LabelName labelName,
+ String comment,
+ UserIdentity createUser,
+ JobExecutionConfiguration jobConfig,
+ Long createTimeMs,
+ String executeSql) {
+super(jobId, jobName, jobStatus, labelName.getDbName(), comment,
createUser,
+jobConfig, createTimeMs, executeSql, null);
+this.dbId = ConnectContext.get().getCurrentDbId();
+this.labelName = labelName.getLabelName();
+}
-// max save task num, do we need to config it?
-private static final int MAX_SAVE_TASK_NUM = 100;
+public InsertJob(ConnectContext ctx,
+ StmtExecutor executor,
+ String labelName,
+ List plans,
+ Set sinkTableNames,
+ Map properties,
+ String comment,
+ JobExecutionConfiguration jobConfig) {
+super(Env.
Re: [PR] [fix](Export) Fix an export error when lower_case_table_names=1 [doris]
BePPPower commented on PR #28389: URL: https://github.com/apache/doris/pull/28389#issuecomment-1857477348 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Branch2.0](fix) Fix an `export` error when lower case table names=1 [doris]
BePPPower commented on PR #28388: URL: https://github.com/apache/doris/pull/28388#issuecomment-1857477641 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refact](fe) Split `SchemaChangeHandler.process` into several function [doris]
SWJTU-ZhangLei commented on PR #28366: URL: https://github.com/apache/doris/pull/28366#issuecomment-1857477385 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Draft](HiveCatalog) Support for querying relational databases under HMS [doris]
BePPPower closed pull request #27737: [Draft](HiveCatalog) Support for querying relational databases under HMS URL: https://github.com/apache/doris/pull/27737 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-flink-connector) branch master updated: [step4][Improve]Format suppressions files and pom files (#274)
This is an automated email from the ASF dual-hosted git repository.
diwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-flink-connector.git
The following commit(s) were added to refs/heads/master by this push:
new 124879d [step4][Improve]Format suppressions files and pom files (#274)
124879d is described below
commit 124879d5b8312729f8f6c834a1c3ea65b48fac32
Author: wudongliang <46414265+donglian...@users.noreply.github.com>
AuthorDate: Fri Dec 15 16:31:31 2023 +0800
[step4][Improve]Format suppressions files and pom files (#274)
---
flink-doris-connector/pom.xml | 40 +---
tools/maven/checkstyle.xml| 10 +-
tools/maven/suppressions.xml | 3 ---
3 files changed, 30 insertions(+), 23 deletions(-)
diff --git a/flink-doris-connector/pom.xml b/flink-doris-connector/pom.xml
index 2d7c3fe..76156ad 100644
--- a/flink-doris-connector/pom.xml
+++ b/flink-doris-connector/pom.xml
@@ -77,12 +77,22 @@ under the License.
3.3.0
3.2.1
UTF-8
-2.18.0
github
1.0.0
8.14
2.17
2.4.2
+4.5.13
+1.15
+4.1.77.Final
+2.13.3
+31.1-jre
+1.7.25
+4.2.0
+1.17.6
+5.10.1
+4.11
+1.3
@@ -146,7 +156,7 @@ under the License.
org.apache.httpcomponents
httpclient
-4.5.13
+${httpcomponents.version}
commons-codec
@@ -158,7 +168,7 @@ under the License.
commons-codec
commons-codec
-1.15
+${commons-codec.version}
@@ -195,37 +205,37 @@ under the License.
io.netty
netty-common
-4.1.77.Final
+${netty.version}
com.fasterxml.jackson.core
jackson-annotations
-2.13.3
+${fasterxml.version}
com.fasterxml.jackson.core
jackson-core
-2.13.3
+${fasterxml.version}
com.fasterxml.jackson.core
jackson-databind
-2.13.3
+${fasterxml.version}
com.google.guava
guava
-31.1-jre
+${guava.version}
org.slf4j
slf4j-log4j12
-1.7.25
+${slf4j.version}
provided
@@ -287,39 +297,39 @@ under the License.
org.hamcrest
hamcrest-core
-1.3
+${hamcrest.version}
test
org.mockito
mockito-core
-4.2.0
+${mockito.version}
test
org.mockito
mockito-inline
-4.2.0
+${mockito.version}
test
junit
junit
-4.11
+${junit.version}
test
org.junit.jupiter
junit-jupiter-api
-5.10.1
+${junit-jupiter.version}
test
org.testcontainers
testcontainers
-1.17.6
+${testcontainers.version}
test
diff --git a/tools/maven/checkstyle.xml b/tools/maven/checkstyle.xml
index a3d8fa3..880563e 100644
--- a/tools/maven/checkstyle.xml
+++ b/tools/maven/checkstyle.xml
@@ -180,11 +180,11 @@ This file is based on the checkstyle file of Apache Beam.
-
-
- https://flink.apache.org/how-to-contribute/code-style-and-quality-common/#avoid-mockito---use-reusable-test-implementations."/>
-
+
+
+
+
+
diff --git a/tools/maven/suppressions.xml b/tools/maven/suppressions.xml
index 2577e5f..6a0417d 100644
--- a/tools/maven/suppressions.xml
+++ b/tools/maven/suppressions.xml
@@ -26,7 +26,4 @@ under the License.
-
-
-
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [step4][Improve]Format suppressions files and pom files [doris-flink-connector]
JNSimba merged PR #274: URL: https://github.com/apache/doris-flink-connector/pull/274 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](executor)make scan task wait timeout config [doris]
doris-robot commented on PR #28467: URL: https://github.com/apache/doris/pull/28467#issuecomment-1857490182 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.1 seconds stream load tsv: 593 seconds loaded 74807831229 Bytes, about 120 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 28.2 seconds inserted 1000 Rows, about 354K ops/s storage size: 17224297128 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](Load)(step2)support nereids load job schedule [doris]
wsjz commented on code in PR #26356:
URL: https://github.com/apache/doris/pull/26356#discussion_r1427703299
##
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/load/LoadMgr.java:
##
@@ -0,0 +1,340 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.jobs.load;
+
+import org.apache.doris.analysis.CompoundPredicate;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.CaseSensibility;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.LabelAlreadyUsedException;
+import org.apache.doris.common.PatternMatcher;
+import org.apache.doris.common.PatternMatcherWrapper;
+import org.apache.doris.common.util.LogBuilder;
+import org.apache.doris.common.util.LogKey;
+import org.apache.doris.job.base.JobExecuteType;
+import org.apache.doris.job.base.JobExecutionConfiguration;
+import org.apache.doris.job.common.JobStatus;
+import org.apache.doris.job.exception.JobException;
+import org.apache.doris.job.extensions.insert.InsertJob;
+import org.apache.doris.job.manager.JobManager;
+import org.apache.doris.load.loadv2.JobState;
+import org.apache.doris.nereids.jobs.load.replay.ReplayLoadLog;
+
+import com.google.common.base.Strings;
+import com.google.common.collect.Lists;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.stream.Collectors;
+
+/**
+ * load manager
+ */
+public class LoadMgr {
+private static final Logger LOG = LogManager.getLogger(LoadMgr.class);
+private Map loadIdToJob = new HashMap<>();
+private final Map>>
dbIdToLabelToLoadJobs = new ConcurrentHashMap<>();
+
+// lock for export job
+// lock is private and must use after db lock
+private final ReentrantReadWriteLock lock = new
ReentrantReadWriteLock(true);
+
+private void readLock() {
+lock.readLock().lock();
+}
+
+private void readUnlock() {
+lock.readLock().unlock();
+}
+
+private void writeLock() {
+lock.writeLock().lock();
+}
+
+private void writeUnlock() {
+lock.writeLock().unlock();
+}
+
+private JobManager getJobManager() {
+return Env.getCurrentEnv().getJobManager();
+}
+
+/**
+ * add load job and add tasks
+ * @param loadJob job
+ */
+public void addLoadJob(InsertJob loadJob) throws DdlException {
+writeLock();
+try {
+Map> labelToLoadJobs =
dbIdToLabelToLoadJobs.get(loadJob.getDbId());
+if (labelToLoadJobs != null &&
labelToLoadJobs.containsKey(loadJob.getLabel())) {
+throw new LabelAlreadyUsedException(loadJob.getLabel());
+}
+unprotectAddJob(loadJob);
+} catch (LabelAlreadyUsedException e) {
+throw new RuntimeException(e);
+} finally {
+writeUnlock();
+}
+}
+
+private void unprotectAddJob(InsertJob job) throws DdlException {
+loadIdToJob.put(job.getJobId(), job);
+try {
+getJobManager().registerJob(job);
+if (!dbIdToLabelToLoadJobs.containsKey(job.getDbId())) {
+dbIdToLabelToLoadJobs.put(job.getDbId(), new
ConcurrentHashMap<>());
+}
+Map> labelToLoadJobs =
dbIdToLabelToLoadJobs.get(job.getDbId());
+if (!labelToLoadJobs.containsKey(job.getLabel())) {
+labelToLoadJobs.put(job.getLabel(), new ArrayList<>());
+}
+labelToLoadJobs.get(job.getLabel()).add(job);
+} catch (org.apache.doris.job.exception.JobException e) {
+throw new DdlException(e.getMessage(), e);
+}
+}
+
+/**
+ * replay load job
+ * @param replayLoadLog load log
+ * @throws DdlException ex
+ */
+public void replayLoadJob(Repl
Re: [PR] [feature](Load)(step2)support nereids load job schedule [doris]
wsjz commented on code in PR #26356:
URL: https://github.com/apache/doris/pull/26356#discussion_r1427703299
##
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/load/LoadMgr.java:
##
@@ -0,0 +1,340 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.jobs.load;
+
+import org.apache.doris.analysis.CompoundPredicate;
+import org.apache.doris.catalog.Database;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.CaseSensibility;
+import org.apache.doris.common.DdlException;
+import org.apache.doris.common.LabelAlreadyUsedException;
+import org.apache.doris.common.PatternMatcher;
+import org.apache.doris.common.PatternMatcherWrapper;
+import org.apache.doris.common.util.LogBuilder;
+import org.apache.doris.common.util.LogKey;
+import org.apache.doris.job.base.JobExecuteType;
+import org.apache.doris.job.base.JobExecutionConfiguration;
+import org.apache.doris.job.common.JobStatus;
+import org.apache.doris.job.exception.JobException;
+import org.apache.doris.job.extensions.insert.InsertJob;
+import org.apache.doris.job.manager.JobManager;
+import org.apache.doris.load.loadv2.JobState;
+import org.apache.doris.nereids.jobs.load.replay.ReplayLoadLog;
+
+import com.google.common.base.Strings;
+import com.google.common.collect.Lists;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.locks.ReentrantReadWriteLock;
+import java.util.stream.Collectors;
+
+/**
+ * load manager
+ */
+public class LoadMgr {
+private static final Logger LOG = LogManager.getLogger(LoadMgr.class);
+private Map loadIdToJob = new HashMap<>();
+private final Map>>
dbIdToLabelToLoadJobs = new ConcurrentHashMap<>();
+
+// lock for export job
+// lock is private and must use after db lock
+private final ReentrantReadWriteLock lock = new
ReentrantReadWriteLock(true);
+
+private void readLock() {
+lock.readLock().lock();
+}
+
+private void readUnlock() {
+lock.readLock().unlock();
+}
+
+private void writeLock() {
+lock.writeLock().lock();
+}
+
+private void writeUnlock() {
+lock.writeLock().unlock();
+}
+
+private JobManager getJobManager() {
+return Env.getCurrentEnv().getJobManager();
+}
+
+/**
+ * add load job and add tasks
+ * @param loadJob job
+ */
+public void addLoadJob(InsertJob loadJob) throws DdlException {
+writeLock();
+try {
+Map> labelToLoadJobs =
dbIdToLabelToLoadJobs.get(loadJob.getDbId());
+if (labelToLoadJobs != null &&
labelToLoadJobs.containsKey(loadJob.getLabel())) {
+throw new LabelAlreadyUsedException(loadJob.getLabel());
+}
+unprotectAddJob(loadJob);
+} catch (LabelAlreadyUsedException e) {
+throw new RuntimeException(e);
+} finally {
+writeUnlock();
+}
+}
+
+private void unprotectAddJob(InsertJob job) throws DdlException {
+loadIdToJob.put(job.getJobId(), job);
+try {
+getJobManager().registerJob(job);
+if (!dbIdToLabelToLoadJobs.containsKey(job.getDbId())) {
+dbIdToLabelToLoadJobs.put(job.getDbId(), new
ConcurrentHashMap<>());
+}
+Map> labelToLoadJobs =
dbIdToLabelToLoadJobs.get(job.getDbId());
+if (!labelToLoadJobs.containsKey(job.getLabel())) {
+labelToLoadJobs.put(job.getLabel(), new ArrayList<>());
+}
+labelToLoadJobs.get(job.getLabel()).add(job);
+} catch (org.apache.doris.job.exception.JobException e) {
+throw new DdlException(e.getMessage(), e);
+}
+}
+
+/**
+ * replay load job
+ * @param replayLoadLog load log
+ * @throws DdlException ex
+ */
+public void replayLoadJob(Repl
Re: [PR] [fix](full compaction) Full compaction should hold meta lock when modifying tablet's meta data [doris]
doris-robot commented on PR #28449: URL: https://github.com/apache/doris/pull/28449#issuecomment-1857498338 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.82 seconds stream load tsv: 606 seconds loaded 74807831229 Bytes, about 117 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 28.5 seconds inserted 1000 Rows, about 350K ops/s storage size: 17219681565 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](hint) query fail print tablet detail info [doris]
yujun777 commented on PR #28476: URL: https://github.com/apache/doris/pull/28476#issuecomment-1857501074 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [cherry-pick](udf) java-udf function open failed cause BE core dump #28063 [doris]
doris-robot commented on PR #28462: URL: https://github.com/apache/doris/pull/28462#issuecomment-1857503878 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 51.71 seconds stream load tsv: 601 seconds loaded 74807831229 Bytes, about 118 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 29.6 seconds inserted 1000 Rows, about 337K ops/s storage size: 17162199419 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore] Add bvar for meta operations of BE [doris]
github-actions[bot] commented on PR #28374: URL: https://github.com/apache/doris/pull/28374#issuecomment-1857504494 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block [doris]
doris-robot commented on PR #27943: URL: https://github.com/apache/doris/pull/27943#issuecomment-1857511066 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.76 seconds stream load tsv: 590 seconds loaded 74807831229 Bytes, about 120 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 28.6 seconds inserted 1000 Rows, about 349K ops/s storage size: 17224236887 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore] Add bvar for meta operations of BE [doris]
gavinchou commented on PR #28374: URL: https://github.com/apache/doris/pull/28374#issuecomment-1857511367 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](proxy) Fix getProxy frequently parse hostname to ip [doris]
xinyiZzz opened a new pull request, #28479: URL: https://github.com/apache/doris/pull/28479 ## Proposed changes Each RPC between Doris FE and BE will call getProxy. Each getProxy will parse hostname and check whether the IP has changed. When the hostname resolution server is unstable, frequent parse hostname will often fail and cause performance problems. Therefore, only check whether the backend IP changes during heartbeat, and if so, recreate the rpc client. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [regression](memtable) add case for memtable flush error status handle [doris]
doris-robot commented on PR #28285: URL: https://github.com/apache/doris/pull/28285#issuecomment-1857511791 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.8 seconds stream load tsv: 605 seconds loaded 74807831229 Bytes, about 117 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.6 seconds inserted 1000 Rows, about 349K ops/s storage size: 17220995675 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](Nereids): put or expansion rule in rewriter [doris]
XieJiann commented on PR #28348: URL: https://github.com/apache/doris/pull/28348#issuecomment-1857516949 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Export) Fix an export error when lower_case_table_names=1 [doris]
doris-robot commented on PR #28389: URL: https://github.com/apache/doris/pull/28389#issuecomment-1857520139 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 43.99 seconds stream load tsv: 582 seconds loaded 74807831229 Bytes, about 122 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17222374405 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore] Add bvar for meta operations of BE [doris]
github-actions[bot] commented on PR #28374: URL: https://github.com/apache/doris/pull/28374#issuecomment-1857521564 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refact](fe) Refact `SchemaChangeHandler.createJob` [doris]
doris-robot commented on PR #28366: URL: https://github.com/apache/doris/pull/28366#issuecomment-1857524559 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.45 seconds stream load tsv: 588 seconds loaded 74807831229 Bytes, about 121 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.2 seconds inserted 1000 Rows, about 342K ops/s storage size: 17222681033 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](proxy) Fix getProxy frequently parse hostname to ip [doris]
github-actions[bot] commented on PR #28479: URL: https://github.com/apache/doris/pull/28479#issuecomment-1857524013 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](iterator) Fix mem leak when initial iterator failed [doris]
w41ter opened a new pull request, #28480: URL: https://github.com/apache/doris/pull/28480 ## Proposed changes Issue Number: close #xxx Memory leaks are prone to occur when using raw pointers, such as directly using the `RETURN_IF_ERROR` macro to skip error handling. This PR replaces the raw pointers managed in `VerticalMergeIterator` with smart pointers; it also fixes some unnecessary copy construction. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](iterator) Fix mem leak when initial iterator failed [doris]
w41ter commented on PR #28480: URL: https://github.com/apache/doris/pull/28480#issuecomment-1857526960 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Test](Job)Add test case [doris]
CalvinKirs opened a new pull request, #28481: URL: https://github.com/apache/doris/pull/28481 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](iterator) Fix mem leak when initial iterator failed [doris]
github-actions[bot] commented on PR #28480: URL: https://github.com/apache/doris/pull/28480#issuecomment-1857535385 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](mtmv)mtmv support partition [doris]
zddr commented on PR #28144: URL: https://github.com/apache/doris/pull/28144#issuecomment-1857537083 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](view) fix npe on create view with comment [doris]
github-actions[bot] commented on PR #28464: URL: https://github.com/apache/doris/pull/28464#issuecomment-1857537310 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](view) fix npe on create view with comment [doris]
github-actions[bot] commented on PR #28464: URL: https://github.com/apache/doris/pull/28464#issuecomment-1857537352 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](publish) Publish task not wait when be's publish task a… [doris]
deardeng commented on code in PR #28079:
URL: https://github.com/apache/doris/pull/28079#discussion_r1427740924
##
fe/fe-core/src/main/java/org/apache/doris/transaction/PublishVersionDaemon.java:
##
@@ -132,27 +133,45 @@ private void publishVersion() {
Map tableIdToTotalDeltaNumRows = Maps.newHashMap();
// try to finish the transaction, if failed just retry in next loop
for (TransactionState transactionState : readyTransactionStates) {
-Stream publishVersionTaskStream =
transactionState
-.getPublishVersionTasks()
-.values()
-.stream()
-.peek(task -> {
-if (task.isFinished() &&
CollectionUtils.isEmpty(task.getErrorTablets())) {
-Map tableIdToDeltaNumRows =
-task.getTableIdToDeltaNumRows();
-tableIdToDeltaNumRows.forEach((tableId, numRows)
-> {
-tableIdToTotalDeltaNumRows
-.computeIfPresent(tableId, (id,
orgNumRows) -> orgNumRows + numRows);
-
tableIdToTotalDeltaNumRows.putIfAbsent(tableId, numRows);
-});
-}
-});
-boolean hasBackendAliveAndUnfinishedTask = publishVersionTaskStream
-.anyMatch(task -> !task.isFinished() &&
infoService.checkBackendAlive(task.getBackendId()));
+AtomicReference finishNum = new AtomicReference<>(0L);
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](hint) query fail print tablet detail info [doris]
doris-robot commented on PR #28476: URL: https://github.com/apache/doris/pull/28476#issuecomment-1857544476 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.32 seconds stream load tsv: 584 seconds loaded 74807831229 Bytes, about 122 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17221189439 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](operation_node)Replace structure to ensure same colRef can be save [doris]
github-actions[bot] commented on PR #28440: URL: https://github.com/apache/doris/pull/28440#issuecomment-1857546288 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [cherry-pick](udf) java-udf function open failed cause BE core dump #28063 [doris]
doris-robot commented on PR #28462: URL: https://github.com/apache/doris/pull/28462#issuecomment-1857547431 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 50.08 seconds stream load tsv: 599 seconds loaded 74807831229 Bytes, about 119 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.8 seconds inserted 1000 Rows, about 335K ops/s storage size: 17162522822 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Bug](memory) Fix exception-unsafe in aggregation node [doris]
xy720 opened a new pull request, #28483: URL: https://github.com/apache/doris/pull/28483 ## Proposed changes Issue Number: close #xxx The alloc function may throw std::bad_alloc exception when the process memory exceed limit. be.INFO: ``` W1214 09:14:17.434849 771103 mem_tracker_limiter.cpp:204] Memory limit exceeded:, process memory used 20.41 GB exceed limit 18.76 GB or sys mem available 9.04 GB less than low water mark 1.60 GB, failed alloc siz e 1.86 MB>, executing msg:>. backend xx.x.x.xxx process memory used 20.41 GB, limit 18.76 GB. If query tracker exceed, set ex ec_mem_limit=8G to change limit, details see be.INFO. Process Memory Summary: OS physical memory 31.26 GB. Process memory usage 20.41 GB, limit 18.76 GB, soft limit 16.88 GB. Sys available memory 9.04 GB, low wa ter mark 1.60 GB, warning water mark 3.20 GB. Refresh interval memory growth 0 B Alloc Stacktrace: @ 0x555cd858bee9 doris::MemTrackerLimiter::print_log_usage() @ 0x555cd859a384 doris::ThreadMemTrackerMgr::exceeded() @ 0x555cd85a0ac4 malloc @ 0x555cd8fcf368 Allocator<>::alloc() @ 0x555cd8fdbdaf doris::vectorized::Arena::add_chunk() @ 0x555cd96dc0ab doris::vectorized::AggregateDataContainer::_expand() @ 0x555cd96aded8 (unknown) @ 0x555cd969fa2c doris::vectorized::AggregationNode::_pre_agg_with_serialized_key() @ 0x555cd96d1d61 std::_Function_handler<>::_M_invoke() @ 0x555cd967ab0b doris::vectorized::AggregationNode::get_next() @ 0x555cd81282a6 doris::ExecNode::get_next_after_projects() @ 0x555cd8452968 doris::PlanFragmentExecutor::get_vectorized_internal() @ 0x555cd845553b doris::PlanFragmentExecutor::open_vectorized_internal() @ 0x555cd8456a9e doris::PlanFragmentExecutor::open() @ 0x555cd842f200 doris::FragmentExecState::execute() @ 0x555cd843280e doris::FragmentMgr::_exec_actual() @ 0x555cd8432d42 _ZNSt17_Function_handlerIFvvEZN5doris11FragmentMgr18exec_plan_fragmentERKNS1_23TExecPlanFragmentParamsESt8funct ionIFvPNS1_20PlanFragmentExecutorUlvE_E9_M_invokeERKSt9_Any_data @ 0x555cd86ead05 doris::ThreadPool::dispatch_thread() @ 0x555cd86e015f doris::Thread::supervise_thread() @ 0x7f3321593ea5 start_thread @ 0x7f33218a69fd __clone @ (nil) (unknown) Memory Tracker Summary:MemTrackerLimiter Label=Load#Id=28448230da1f432e-8a66597e10329235, Type=load, Limit=8.00 GB(8589934592 B), Use d=273.64 MB(286932828 B), Peak=273.98 MB(287286184 B) MemTracker Label=VDataStreamRecvr:28448230da1f432e-8a66597e1032923a, Parent Label=Load#Id=28448230da1f432e-8a66597e10329235, Used=1.0 8 KB(1104 B), Peak=2.66 KB(2728 B) MemTracker Label=OlapTablePartitionParam, Parent Label=Load#Id=28448230da1f432e-8a66597e10329235, Used=533.00 B(533 B), Peak=533.00 B (533 B) MemTracker Label=OlapTableSink:0, Parent Label=Load#Id=28448230da1f432e-8a66597e10329235, Used=23.89 KB(24464 B), Peak=23.89 KB(24464 B) MemTracker Label=IndexChannel:indexID=13928, Parent Label=Load#Id=28448230da1f432e-8a66597e10329235, Used=18.74 KB(19192 B), Peak=18. 74 KB(19192 B) MemTracker Label=NodeChannel:indexID=13928:threadId=139848762504960, Parent Label=Load#Id=28448230da1f432e-8a66597e10329235, Used=216 .00 B(216 B), Peak=216.00 B(216 B) MemTracker Label=NodeChannel:indexID=13928:threadId=139848762504960, Parent Label=Load#Id=28448230da1f432e-8a66597e10329235, Used=216 .00 B(216 B), Peak=216.00 B(216 B) MemTracker Label=NodeChannel:indexID=13928:threadId=139848762504960, Parent Label=Load#Id=28448230da1f432e-8a66597e10329235, Used=216.00 B(216 B), Peak=216.00 B(216 B) MemTracker Label=VDataStreamRecvr:28448230da1f432e-8a66597e10329236, Parent Label=Load#Id=28448230da1f432e-8a66597e10329235, Used=75.12 KB(76920 B), Peak=3.34 MB(3498744 B) MemTracker Label=AggregationNode:Data, Parent Label=Load#Id=28448230da1f432e-8a66597e10329235, Used=699.98 KB(716776 B), Peak=699.98 KB(716776 B) MemTracker Label=VDataStreamSender:28448230da1f432e-8a66597e10329236, Parent Label=Load#Id=28448230da1f432e-8a66597e10329235, Used=3.44 KB(3520 B), Peak=588.59 KB(602720 B) MemTracker Label=VDataStreamRecvr:28448230da1f432e-8a66597e10329239, Parent Label=Load#Id=28448230da1f432e-8a66597e10329235, Used=5.46 KB(5592 B), Peak=1.57 MB(1645560 B) MemTracker Label=AggregationNode:Data, Parent Label=Load#Id=28448230da1f432e-8a66597e10329235, Used=20.00 MB(20975576 B), Peak=20.00 MB(20975576 B) MemTracker Label=VDataStreamRecvr:28448230da1f432e-8a66597e10329238, Parent Label=Load#Id=28448230da1f432e-8a66597e10329235, Used=223.35 KB(228712 B), Peak=6.88 MB(7219304 B) MemTracker Label=AggregationNode:Data, Parent Label=Load#Id
Re: [PR] [docker][regregression]update routine load cases [doris]
XuJianxu commented on PR #28450: URL: https://github.com/apache/doris/pull/28450#issuecomment-1857552165 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](memory) Fix exception-unsafe in aggregation node [doris]
xy720 commented on PR #28483: URL: https://github.com/apache/doris/pull/28483#issuecomment-1857553594 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](operation_node)Replace structure to ensure same colRef can be save [doris]
zhiqiang- commented on PR #28440: URL: https://github.com/apache/doris/pull/28440#issuecomment-1857556365 @TomaYoko hello, thanks for your contribution, you can see my comment before buildall. `sh run-regression-test --run -s minus_except -getOut` will make help -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [regression](memtable) add case for memtable flush error status handle [doris]
doris-robot commented on PR #28285: URL: https://github.com/apache/doris/pull/28285#issuecomment-1857558409 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.61 seconds stream load tsv: 604 seconds loaded 74807831229 Bytes, about 118 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 28.5 seconds inserted 1000 Rows, about 350K ops/s storage size: 17219539841 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](memory) Fix exception-unsafe in aggregation node [doris]
github-actions[bot] commented on PR #28483: URL: https://github.com/apache/doris/pull/28483#issuecomment-1857560157 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Feature](Variant) Implement variant new sub column access method [doris]
eldenmoon opened a new pull request, #28484: URL: https://github.com/apache/doris/pull/28484 The query SELECT v["a"]["b"] from simple_var WHERE cast(v["a"]["b"] as int) = 1 encompasses three primary testing scenarios: ``` 1. A basic test involving the variant data type. 2. A scenario dealing with GitHub event data in the context of a variant. 3. A case related to the TPC-H benchmark using a variant. ``` ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](Variant) Implement variant new sub column access method [doris]
eldenmoon commented on PR #28484: URL: https://github.com/apache/doris/pull/28484#issuecomment-1857560671 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](operation_node)Replace structure to ensure same colRef can be save [doris]
zhiqiang- commented on code in PR #28440:
URL: https://github.com/apache/doris/pull/28440#discussion_r1427750874
##
regression-test/data/minus_except_test/minus_except.out:
##
@@ -0,0 +1,3 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !select --
++06-00 +06-00
Review Comment:
incorrect content format
maybe:
`[+06-00, +06-00]`
or you can try
`sh run-regression-test --run -s minus_except -getOut`
##
regression-test/suites/minus_except_test/minus_except.groovy:
##
@@ -24,7 +24,6 @@ suite("minus_except") {
sql "CREATE TABLE ${tableNameDate}(c2 date) DUPLICATE key(c2) DISTRIBUTED
BY hash(c2) BUCKETS auto PROPERTIES ('replication_num' = '1')"
sql "insert into ${tableNameVarchar} values('+06-00');"
sql "insert into ${tableNameDate} values('1990-11-11');"
-qt_select " SELECT c1,c1 FROM ${tableNameVarchar} MINUS SELECT c2,c2 FROM
${tableNameDate};"
Review Comment:
This test should be added back, change the result format PLS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [branch-1.2](memory) Fix AggFunc memory leak due to incorrect destroy [doris]
xy720 opened a new pull request, #28485: URL: https://github.com/apache/doris/pull/28485 ## Proposed changes cherry-pick #19126 into branch-1.2 ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](hint) query fail print tablet detail info [doris]
yujun777 commented on PR #28476: URL: https://github.com/apache/doris/pull/28476#issuecomment-1857563235 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Test](Job)Add test case [doris]
CalvinKirs commented on PR #28481: URL: https://github.com/apache/doris/pull/28481#issuecomment-1857561097 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](operation_node)Replace structure to ensure same colRef can be save [doris]
zhiqiang- commented on code in PR #28440: URL: https://github.com/apache/doris/pull/28440#discussion_r1427750874 ## regression-test/data/minus_except_test/minus_except.out: ## @@ -0,0 +1,3 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !select -- ++06-00 +06-00 Review Comment: incorrect content format maybe: `[+06-00, +06-00]` or you can try `sh run-regression-test --run -s minus_except -genOut` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](mtmv)mtmv support partition [doris]
zddr commented on PR #28144: URL: https://github.com/apache/doris/pull/28144#issuecomment-1857566431 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](iterator) Fix mem leak when initial iterator failed [doris]
github-actions[bot] commented on PR #28480: URL: https://github.com/apache/doris/pull/28480#issuecomment-1857566859 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](iterator) Fix mem leak when initial iterator failed [doris]
github-actions[bot] commented on PR #28480: URL: https://github.com/apache/doris/pull/28480#issuecomment-1857566910 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [feature](Nereids) elimite join by foereign key [doris]
XieJiann opened a new pull request, #28486: URL: https://github.com/apache/doris/pull/28486 ## Proposed changes eliminate join by foreign key ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](iterator) Fix mem leak when initial iterator failed [doris]
doris-robot commented on PR #28480: URL: https://github.com/apache/doris/pull/28480#issuecomment-1857568153 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.06 seconds stream load tsv: 587 seconds loaded 74807831229 Bytes, about 121 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 29.0 seconds inserted 1000 Rows, about 344K ops/s storage size: 17224171547 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Chore](Workflow)Fix auto-reply not work [doris]
CalvinKirs opened a new pull request, #28487: URL: https://github.com/apache/doris/pull/28487 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Yukang-Lian commented on PR #27726: URL: https://github.com/apache/doris/pull/27726#issuecomment-1857569619 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Mryange opened a new pull request, #28488:
URL: https://github.com/apache/doris/pull/28488
## Proposed changes
core in pipelineX
```
vdata_stream_recvr.h:272] Check failed: _is_cancelled ||
!_block_queue.empty() || _num_remaining_senders == 0 _is_cancelled: 0,
_block_queue_empty: 1, _num_remaining_senders: 2
```
```
bool VDataStreamRecvr::ready_to_read() {
for (const auto& queue : _sender_queues) {
if (queue->should_wait()) {
return false;
}
}
return true;
}
bool VDataStreamRecvr::SenderQueue::should_wait() {
std::unique_lock l(_lock);
return !_is_cancelled && _block_queue.empty() && _num_remaining_senders
> 0;
}
```
However, pipelineX is different, so here it is changed to be consistent.
## Further comments
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](session variables) Make default value of max_execution_time same to query_timeout [doris]
github-actions[bot] commented on PR #28474: URL: https://github.com/apache/doris/pull/28474#issuecomment-1857570639 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](session variables) Make default value of max_execution_time same to query_timeout [doris]
github-actions[bot] commented on PR #28474: URL: https://github.com/apache/doris/pull/28474#issuecomment-1857570709 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] FE 频繁打满64G内存导致宕机，集群上只有Broker Load在定时执行，过一段时间内存就满了 [doris]
ziyanTOP commented on issue #27594: URL: https://github.com/apache/doris/issues/27594#issuecomment-1857572219 一样的问题，minor gc的频率跟不上老年代增长的速度，最后三个fe节点全部查询排队超时卡死宕机，建议用prometheus+grafana监控fe的JVM，看看到底问题出在哪，顺便改下你的参数，年轻代等于老年代的1/3，并且不要用-XX:NewRatio=3这种，而是固定设置成-Xmn16G，打开CMS的并行重标记，不然minor gc那点时间这么多内存根本标记不完，然后调低CMS初始化时的内存占比，80%太靠后了，可能gc没完成服务就down了，可以改成60或者65，实测有效，我的集群调整完至今没有fe宕机 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](operation_node)Replace structure to ensure same colRef can be save [doris]
TomaYoko commented on code in PR #28440: URL: https://github.com/apache/doris/pull/28440#discussion_r1427768936 ## regression-test/data/minus_except_test/minus_except.out: ## @@ -0,0 +1,3 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !select -- ++06-00 +06-00 Review Comment: This out file is generated by genOut command( I use the version of 2.0.2 ) and local test has passed , shall I change the version to master and regenerate the out file? What I wonder is why the out file genterated by local which use command below can't passed by remote command I used  Looking forward to your answer @zhiqiang- -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature](Variant) Implement variant new sub column access method [doris]
eldenmoon commented on PR #28484: URL: https://github.com/apache/doris/pull/28484#issuecomment-1857574785 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Chore](Workflow)Fix auto-reply not work [doris]
CalvinKirs closed pull request #28487: [Chore](Workflow)Fix auto-reply not work URL: https://github.com/apache/doris/pull/28487 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](decimalv2) support check overflow for decimalv2 arithmetics [doris]
github-actions[bot] commented on PR #28456: URL: https://github.com/apache/doris/pull/28456#issuecomment-1857579154 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
github-actions[bot] commented on code in PR #28488:
URL: https://github.com/apache/doris/pull/28488#discussion_r1427773381
##
be/src/vec/runtime/vdata_stream_recvr.cpp:
##
@@ -122,6 +122,16 @@ Status
VDataStreamRecvr::SenderQueue::_inner_get_batch_without_lock(Block* block
return Status::OK();
}
+void VDataStreamRecvr::SenderQueue::try_set_dep_ready_without_lock() {
Review Comment:
warning: method 'try_set_dep_ready_without_lock' can be made static
[readability-convert-member-functions-to-static]
be/src/vec/runtime/vdata_stream_recvr.h:242:
```diff
- void try_set_dep_ready_without_lock();
+ static void try_set_dep_ready_without_lock();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](decimalv2) support check overflow for decimalv2 arithmetics [doris]
github-actions[bot] commented on PR #28456: URL: https://github.com/apache/doris/pull/28456#issuecomment-1857579216 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](operation_node)Replace structure to ensure same colRef can be save [doris]
TomaYoko commented on code in PR #28440:
URL: https://github.com/apache/doris/pull/28440#discussion_r142353
##
regression-test/suites/minus_except_test/minus_except.groovy:
##
@@ -24,7 +24,6 @@ suite("minus_except") {
sql "CREATE TABLE ${tableNameDate}(c2 date) DUPLICATE key(c2) DISTRIBUTED
BY hash(c2) BUCKETS auto PROPERTIES ('replication_num' = '1')"
sql "insert into ${tableNameVarchar} values('+06-00');"
sql "insert into ${tableNameDate} values('1990-11-11');"
-qt_select " SELECT c1,c1 FROM ${tableNameVarchar} MINUS SELECT c2,c2 FROM
${tableNameDate};"
Review Comment:
> This test should be added back, change the result format PLS.
I added this test before,but it seems can't pass on the remote which reason
may be due to grammar failure（the test can be tested successfully locallly）
I wonder if is the same reason caused like the one replied above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [improvement](decimalv2) support check overflow for decimalv2 arithmetics (#28456) [doris]
jacktengg opened a new pull request, #28489: URL: https://github.com/apache/doris/pull/28489 ## Proposed changes Pick from #28456 ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](decimalv2) support check overflow for decimalv2 arithmetics (#28456) [doris]
github-actions[bot] commented on PR #28489: URL: https://github.com/apache/doris/pull/28489#issuecomment-1857595574 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](decimalv2) support check overflow for decimalv2 arithmetics [doris]
jacktengg commented on PR #28456: URL: https://github.com/apache/doris/pull/28456#issuecomment-1857596171 run pipelinex_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refact](fe) Refact `SchemaChangeHandler.createJob` [doris]
SWJTU-ZhangLei commented on PR #28366: URL: https://github.com/apache/doris/pull/28366#issuecomment-1857596520 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](coverage): refresh the coverage file before exiting the program [doris]
github-actions[bot] commented on PR #28354: URL: https://github.com/apache/doris/pull/28354#issuecomment-1857597226 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](memory) Fix exception-unsafe in aggregation node [doris]
github-actions[bot] commented on PR #28483: URL: https://github.com/apache/doris/pull/28483#issuecomment-1857598774 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore] Add bvar for meta operations of BE [doris]
doris-robot commented on PR #28374: URL: https://github.com/apache/doris/pull/28374#issuecomment-1857600425 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.62 seconds stream load tsv: 592 seconds loaded 74807831229 Bytes, about 120 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.0 seconds inserted 1000 Rows, about 344K ops/s storage size: 17219968263 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
