(doris) branch master updated: [fix](export) fix potential export concurrency issue (#43109)
This is an automated email from the ASF dual-hosted git repository.
kirs pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 00be5d4a209 [fix](export) fix potential export concurrency issue
(#43109)
00be5d4a209 is described below
commit 00be5d4a2098eef19f9fdb2936f4269a34b56a41
Author: Mingyu Chen (Rayner)
AuthorDate: Sat Nov 2 08:35:44 2024 +0800
[fix](export) fix potential export concurrency issue (#43109)
### What problem does this PR solve?
Problem Summary:
```
2024-11-01 19:42:52,521 WARN (mysql-nio-pool-117|9514)
[StmtExecutor.execute():616] Analyze failed. stmt[250257,
59c581a512e7468f-b1cfd7d4b63fed33]
org.apache.doris.common.NereidsException: errCode = 2, detailMessage =
java.util.ConcurrentModificationException
at
org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:780)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:601)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:564)
~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:554)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:340)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:243)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:208)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:236)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:413)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52)
~[doris-fe.jar:1.2-SNAPSHOT]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
~[?:?]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: org.apache.doris.common.AnalysisException: errCode = 2,
detailMessage = java.util.ConcurrentModificationException
... 13 more
Caused by: java.util.ConcurrentModificationException
at java.util.ArrayList.forEach(ArrayList.java:1513) ~[?:?]
at
org.apache.doris.load.ExportMgr.addExportJobAndRegisterTask(ExportMgr.java:120)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.nereids.trees.plans.commands.ExportCommand.run(ExportCommand.java:149)
~[doris-fe.jar:1.2-SNAPSHOT]
at
org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:749)
~[doris-fe.jar:1.2-SNAPSHOT]
... 12 more
```
### Check List (For Committer)
- Test
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [x] Previous test can cover this change.
- [ ] No colde files have been changed.
- [ ] Other reason
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [ ] No.
- [ ] Yes.
- Release note
None
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java| 2 +-
fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java | 4 ++--
fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java | 4 +---
.../org/apache/doris/nereids/trees/plans/commands/ExportCommand.java | 2 +-
4 files changed, 5 insertions(+), 7 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java
index a9ce85b2d3e..ba7aa50ec69 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java
@@ -208,7 +208,7 @@ public class ExportStmt extends StatementBase implements
NotFallbackInParser {
}
private void setJob() throws UserException {
-exportJob = new ExportJob();
+exportJob = new ExportJob(Env.getCurrentEnv().getNextId());
Database db =
Env.getCurrentInternalCatalog().getDbOrDdlException(this.tblName.getDb());
ex
Re: [PR] [fix](export) fix potential export concurrency issue [doris]
CalvinKirs merged PR #43109: URL: https://github.com/apache/doris/pull/43109 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [improvement](statistics)External table getRowCount return -1 when row count is not available or row count is 0. (#43009) (#43087)
This is an automated email from the ASF dual-hosted git repository.
lijibing pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 3b9a2409b49 [improvement](statistics)External table getRowCount return
-1 when row count is not available or row count is 0. (#43009) (#43087)
3b9a2409b49 is described below
commit 3b9a2409b49b5d2c9cb8b6242151c643e4fe5bf1
Author: Jibing-Li <64681310+jibing...@users.noreply.github.com>
AuthorDate: Sat Nov 2 08:38:55 2024 +0800
[improvement](statistics)External table getRowCount return -1 when row
count is not available or row count is 0. (#43009) (#43087)
backport: https://github.com/apache/doris/pull/43009
---
.../java/org/apache/doris/catalog/OlapTable.java | 8 +--
.../main/java/org/apache/doris/catalog/Table.java | 2 +-
.../java/org/apache/doris/catalog/TableIf.java | 2 +
.../doris/catalog/external/ExternalTable.java | 4 +-
.../doris/catalog/external/HMSExternalTable.java | 7 +-
.../catalog/external/PaimonExternalTable.java | 4 +-
.../doris/datasource/ExternalRowCountCache.java| 6 +-
.../doris/external/iceberg/util/IcebergUtils.java | 6 +-
.../doris/statistics/StatisticsAutoCollector.java | 2 +-
.../doris/statistics/util/StatisticsUtil.java | 22 +++---
.../datasource/ExternalRowCountCacheTest.java | 83 +-
11 files changed, 116 insertions(+), 30 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index 3d14b92de07..9b95b1b20ec 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -119,8 +119,6 @@ public class OlapTable extends Table {
WAITING_STABLE
}
-public static long ROW_COUNT_BEFORE_REPORT = -1;
-
private volatile OlapTableState state;
// index id -> index meta
@@ -1298,12 +1296,12 @@ public class OlapTable extends Table {
if (index == null) {
LOG.warn("Index {} not exist in partition {}, table {}, {}",
indexId, entry.getValue().getName(), id, name);
-return ROW_COUNT_BEFORE_REPORT;
+return UNKNOWN_ROW_COUNT;
}
if (strict && !index.getRowCountReported()) {
-return ROW_COUNT_BEFORE_REPORT;
+return UNKNOWN_ROW_COUNT;
}
-rowCount += index.getRowCount() == -1 ? 0 : index.getRowCount();
+rowCount += index.getRowCount() == UNKNOWN_ROW_COUNT ? 0 :
index.getRowCount();
}
return rowCount;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
index 7cbef2a5d6c..17df068ff21 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Table.java
@@ -614,6 +614,6 @@ public abstract class Table extends MetaObject implements
Writable, TableIf {
@Override
public long fetchRowCount() {
-return 0;
+return UNKNOWN_ROW_COUNT;
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
index c1bcf5b2179..4586ed62900 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableIf.java
@@ -45,6 +45,8 @@ import java.util.concurrent.TimeUnit;
public interface TableIf {
Logger LOG = LogManager.getLogger(TableIf.class);
+long UNKNOWN_ROW_COUNT = -1;
+
void readLock();
boolean tryReadLock(long timeout, TimeUnit unit);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/catalog/external/ExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/external/ExternalTable.java
index 90188e18fed..0a8ffce08ce 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/catalog/external/ExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/catalog/external/ExternalTable.java
@@ -290,7 +290,7 @@ public class ExternalTable implements TableIf, Writable,
GsonPostProcessable {
makeSureInitialized();
} catch (Exception e) {
LOG.warn("Failed to initialize table {}.{}.{}", catalog.getName(),
dbName, name, e);
-return 0;
+return TableIf.UNKNOWN_ROW_COUNT;
}
// All external table should get external row count from cache.
return
Env.getCurrentEnv().getExtMetaCacheMgr().getRowCountCache().getCachedRowCount(catalog.getId(),
dbId, id);
@@ -302,7 +302,7 @@ public class ExternalTable implements TableIf, Writable,
GsonPostProcessable {
* This is called by ExternalRowCountCache to lo
Re: [PR] [fix](persist) Fix the generated dropping tmp partition sql [doris]
CalvinKirs merged PR #43079: URL: https://github.com/apache/doris/pull/43079 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](persist) Fix the generated dropping tmp partition sql (#43079)
This is an automated email from the ASF dual-hosted git repository.
kirs pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 9b6bee56b26 [fix](persist) Fix the generated dropping tmp partition
sql (#43079)
9b6bee56b26 is described below
commit 9b6bee56b26ea41c3eb4167a3ab2a666230fbc14
Author: walter
AuthorDate: Sat Nov 2 08:41:10 2024 +0800
[fix](persist) Fix the generated dropping tmp partition sql (#43079)
The syntax is:
```
KW_DROP opt_tmp:isTempPartition KW_PARTITION ...
```
---
.../src/main/java/org/apache/doris/persist/DropPartitionInfo.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/persist/DropPartitionInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/persist/DropPartitionInfo.java
index 0138549405e..de2bd7295c9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/DropPartitionInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/DropPartitionInfo.java
@@ -60,11 +60,11 @@ public class DropPartitionInfo implements Writable {
this.recycleTime = recycleTime;
StringBuilder sb = new StringBuilder();
-sb.append("DROP PARTITION ");
+sb.append("DROP ");
if (isTempPartition) {
sb.append("TEMPORARY ");
}
-sb.append("`").append(partitionName).append("`");
+sb.append("PARTITION `").append(partitionName).append("`");
if (forceDrop) {
sb.append(" FORCE");
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43079-branch-3.0 created (now 757b0ea23c5)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43079-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git at 757b0ea23c5 [fix](regression-test) fix test_hive_serde_prop #42886 (#43099) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43109-branch-3.0 created (now 757b0ea23c5)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43109-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git at 757b0ea23c5 [fix](regression-test) fix test_hive_serde_prop #42886 (#43099) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](statistics)External table getRowCount return -1 when row count is not available or row count is 0. (#43009) [doris]
Jibing-Li merged PR #43087: URL: https://github.com/apache/doris/pull/43087 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](persist) Fix the generated dropping tmp partition sql [doris]
doris-robot commented on PR #43124: URL: https://github.com/apache/doris/pull/43124#issuecomment-2452764678 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](persist) Fix the generated dropping tmp partition sql [doris]
doris-robot commented on PR #43124: URL: https://github.com/apache/doris/pull/43124#issuecomment-2452764652 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43079-branch-3.0 updated (757b0ea23c5 -> 1fd3ff58c4a)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43079-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from 757b0ea23c5 [fix](regression-test) fix test_hive_serde_prop #42886 (#43099) add 1fd3ff58c4a [fix](persist) Fix the generated dropping tmp partition sql (#43079) No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/doris/persist/DropPartitionInfo.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] branch-3.0: [fix](persist) Fix the generated dropping tmp partition sql [doris]
github-actions[bot] opened a new pull request, #43124: URL: https://github.com/apache/doris/pull/43124 PR Body: The syntax is: ``` KW_DROP opt_tmp:isTempPartition KW_PARTITION ... ``` Cherry-picked from #43079 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](export) fix potential export concurrency issue #43109 (#43117)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 9e7a1c9ae8d [fix](export) fix potential export concurrency issue
#43109 (#43117)
9e7a1c9ae8d is described below
commit 9e7a1c9ae8d60c47dfbb4c4d853a04f358fec1e6
Author: Mingyu Chen (Rayner)
AuthorDate: Sat Nov 2 08:53:43 2024 +0800
[fix](export) fix potential export concurrency issue #43109 (#43117)
bp #43109
---
fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java| 2 +-
fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java | 4 ++--
fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java | 4 +---
.../org/apache/doris/nereids/trees/plans/commands/ExportCommand.java | 2 +-
4 files changed, 5 insertions(+), 7 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java
index 855379cbc37..581dff1917b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ExportStmt.java
@@ -208,7 +208,7 @@ public class ExportStmt extends StatementBase {
}
private void setJob() throws UserException {
-exportJob = new ExportJob();
+exportJob = new ExportJob(Env.getCurrentEnv().getNextId());
Database db =
Env.getCurrentInternalCatalog().getDbOrDdlException(this.tblName.getDb());
exportJob.setDbId(db.getId());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
b/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
index 21df83c1192..e04472f43df 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
@@ -397,8 +397,8 @@ public class ExportJob implements Writable {
return statementBase;
}
-public List getTaskExecutors() {
-return jobExecutorList;
+public List getCopiedTaskExecutors() {
+return Lists.newArrayList(jobExecutorList);
}
private void generateExportJobExecutor() {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java
index 2c0a6ec5916..876dcf9def4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/ExportMgr.java
@@ -95,8 +95,6 @@ public class ExportMgr {
}
public void addExportJobAndRegisterTask(ExportJob job) throws Exception {
-long jobId = Env.getCurrentEnv().getNextId();
-job.setId(jobId);
writeLock();
try {
if (dbTolabelToExportJobId.containsKey(job.getDbId())
@@ -113,7 +111,7 @@ public class ExportMgr {
BrokerUtil.deleteDirectoryWithFileSystem(fullPath.substring(0,
fullPath.lastIndexOf('/') + 1),
job.getBrokerDesc());
}
-job.getTaskExecutors().forEach(executor -> {
+job.getCopiedTaskExecutors().forEach(executor -> {
Env.getCurrentEnv().getTransientTaskManager().addMemoryTask(executor);
});
Env.getCurrentEnv().getEditLog().logExportCreate(job);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExportCommand.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExportCommand.java
index 263bf43e355..fd5d604c520 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExportCommand.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExportCommand.java
@@ -241,7 +241,7 @@ public class ExportCommand extends Command implements
ForwardWithSync {
private ExportJob generateExportJob(ConnectContext ctx, Map fileProperties, TableName tblName)
throws UserException {
-ExportJob exportJob = new ExportJob();
+ExportJob exportJob = new ExportJob(Env.getCurrentEnv().getNextId());
// set export job and check catalog/db/table
CatalogIf catalog =
ctx.getEnv().getCatalogMgr().getCatalogOrAnalysisException(tblName.getCtl());
DatabaseIf db = catalog.getDbOrAnalysisException(tblName.getDb());
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](auditlog) add missing audit log fields and duplicate audit log error [doris]
morningman commented on PR #42262: URL: https://github.com/apache/doris/pull/42262#issuecomment-2452772258 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](export) fix potential export concurrency issue #43109 [doris]
morningman merged PR #43117: URL: https://github.com/apache/doris/pull/43117 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](Export) fix dead lock of parallel Exporting (#43097)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new d78dbf3ed7f [fix](Export) fix dead lock of parallel Exporting (#43097)
d78dbf3ed7f is described below
commit d78dbf3ed7fb70009b9a2538fcaf534630ae9a31
Author: Tiewei Fang <43782773+bepppo...@users.noreply.github.com>
AuthorDate: Sat Nov 2 08:54:36 2024 +0800
[fix](Export) fix dead lock of parallel Exporting (#43097)
bp: #43083
---
fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java | 7 +++
fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java | 4 ++--
2 files changed, 5 insertions(+), 6 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
b/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
index 938a9d7711a..63d88de46af 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
@@ -665,7 +665,7 @@ public class ExportJob implements Writable {
}
// we only persist Pending/Cancel/Finish state
if (!isReplay && newState != JobState.IN_QUEUE && newState !=
JobState.EXPORTING) {
-Env.getCurrentEnv().getEditLog().logExportUpdateState(id,
newState);
+Env.getCurrentEnv().getEditLog().logExportUpdateState(this,
newState);
}
return true;
}
@@ -846,10 +846,9 @@ public class ExportJob implements Writable {
}
// used for persisting one log
-public StateTransfer(long jobId, JobState state) {
-this.jobId = jobId;
+public StateTransfer(ExportJob job, JobState state) {
+this.jobId = job.getId();
this.state = state;
-ExportJob job = Env.getCurrentEnv().getExportMgr().getJob(jobId);
this.startTimeMs = job.getStartTimeMs();
this.finishTimeMs = job.getFinishTimeMs();
this.failMsg = job.getFailMsg();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
b/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
index 87b2b746d5e..963a1e06c8a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
@@ -1476,8 +1476,8 @@ public class EditLog {
logEdit(OperationType.OP_EXPORT_CREATE, job);
}
-public void logExportUpdateState(long jobId, ExportJob.JobState newState) {
-ExportJob.StateTransfer transfer = new ExportJob.StateTransfer(jobId,
newState);
+public void logExportUpdateState(ExportJob job, ExportJob.JobState
newState) {
+ExportJob.StateTransfer transfer = new ExportJob.StateTransfer(job,
newState);
logEdit(OperationType.OP_EXPORT_UPDATE_STATE, transfer);
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (399b43751f8 -> 2ad6cb780a5)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 399b43751f8 [fix](Export) fix dead lock of parallel Exporting (#43083) add 2ad6cb780a5 [Fix](PaimonCatalog) fix the problem that paimon catalog can not access to OSS-HDFS (#42585) No new revisions were added by this update. Summary of changes: .../org/apache/doris/common/util/LocationPath.java| 16 ++-- .../datasource/paimon/PaimonFileExternalCatalog.java | 19 +-- .../apache/doris/common/util/LocationPathTest.java| 4 +++- 3 files changed, 30 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](PaimonCatalog) fix the problem that paimon catalog can not access to OSS-HDFS [doris]
morningman merged PR #42585: URL: https://github.com/apache/doris/pull/42585 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Export) fix dead lock of parallel Exporting [doris]
morningman merged PR #43097: URL: https://github.com/apache/doris/pull/43097 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](AuditLog) add `cloudClusterName` field for `audit_log` system table [doris]
morningman closed pull request #42084: [fix](AuditLog) add `cloudClusterName` field for `audit_log` system table URL: https://github.com/apache/doris/pull/42084 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Fix](PaimonCatalog) fix the problem that paimon catalog can not access to OSS-HDFS [doris]
github-actions[bot] commented on PR #43125: URL: https://github.com/apache/doris/pull/43125#issuecomment-2452773812 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [Fix](PaimonCatalog) fix the problem that paimon catalog can not access to OSS-HDFS [doris]
doris-robot commented on PR #43125: URL: https://github.com/apache/doris/pull/43125#issuecomment-2452773828 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](hive)fix after insert overwrite hive table, data error [doris]
morningman commented on PR #43049: URL: https://github.com/apache/doris/pull/43049#issuecomment-2452773916 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-42585-branch-2.1 created (now 9e7a1c9ae8d)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-42585-branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git at 9e7a1c9ae8d [fix](export) fix potential export concurrency issue #43109 (#43117) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](Export) fix dead lock of parallel Exporting (#43083)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 399b43751f8 [fix](Export) fix dead lock of parallel Exporting (#43083)
399b43751f8 is described below
commit 399b43751f8acc4e7fef9c807cc254f66a4df13e
Author: Tiewei Fang <43782773+bepppo...@users.noreply.github.com>
AuthorDate: Sat Nov 2 08:55:27 2024 +0800
[fix](Export) fix dead lock of parallel Exporting (#43083)
### What problem does this PR solve?
Problem Summary:
There are 2 locks, lock of ExportMgr and lock of ExportJob(synchronized
lock).
Previously, the lock order is wrong:
1. When cancelling job, it will first get job lock, then getting mgr
lock.
2. When removing old job, it will first get mgr lock, then getting job
lock.
This PR fix it by always getting job lock after mgr lock, to avoid dead
lock.
---
fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java| 4 ++--
.../src/main/java/org/apache/doris/load/ExportJobStateTransfer.java | 5 ++---
fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java | 4 ++--
3 files changed, 6 insertions(+), 7 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
b/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
index 9841ae2b11d..0a07b5b6851 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
@@ -706,7 +706,7 @@ public class ExportJob implements Writable {
if (FeConstants.runningUnitTest) {
return;
}
-Env.getCurrentEnv().getEditLog().logExportUpdateState(id,
ExportJobState.CANCELLED);
+Env.getCurrentEnv().getEditLog().logExportUpdateState(this,
ExportJobState.CANCELLED);
}
private void exportExportJob() {
@@ -749,7 +749,7 @@ public class ExportJob implements Writable {
outfileInfo = GsonUtils.GSON.toJson(allOutfileInfo);
// Clear the jobExecutorList to release memory.
jobExecutorList.clear();
-Env.getCurrentEnv().getEditLog().logExportUpdateState(id,
ExportJobState.FINISHED);
+Env.getCurrentEnv().getEditLog().logExportUpdateState(this,
ExportJobState.FINISHED);
}
public void replayExportJobState(ExportJobState newState) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/load/ExportJobStateTransfer.java
b/fe/fe-core/src/main/java/org/apache/doris/load/ExportJobStateTransfer.java
index 06253b1f1e8..4b6f2ff9f51 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/ExportJobStateTransfer.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/ExportJobStateTransfer.java
@@ -54,10 +54,9 @@ public class ExportJobStateTransfer implements Writable {
}
// used for persisting one log
-public ExportJobStateTransfer(long jobId, ExportJobState state) {
-this.jobId = jobId;
+public ExportJobStateTransfer(ExportJob job, ExportJobState state) {
+this.jobId = job.getId();
this.state = state;
-ExportJob job = Env.getCurrentEnv().getExportMgr().getJob(jobId);
this.startTimeMs = job.getStartTimeMs();
this.finishTimeMs = job.getFinishTimeMs();
this.failMsg = job.getFailMsg();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
b/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
index 5f94c95a249..83e7977a983 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
@@ -1621,8 +1621,8 @@ public class EditLog {
}
}
-public void logExportUpdateState(long jobId, ExportJobState newState) {
-ExportJobStateTransfer transfer = new ExportJobStateTransfer(jobId,
newState);
+public void logExportUpdateState(ExportJob job, ExportJobState newState) {
+ExportJobStateTransfer transfer = new ExportJobStateTransfer(job,
newState);
logEdit(OperationType.OP_EXPORT_UPDATE_STATE, transfer);
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] Support move truncated old data to recycle bin [doris]
Vallishp commented on PR #43107: URL: https://github.com/apache/doris/pull/43107#issuecomment-2452867248 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](type cast) Fix some implicit cast [doris]
zhiqiang- commented on PR #43050: URL: https://github.com/apache/doris/pull/43050#issuecomment-2452869114 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch asf-site updated (8ff698807d1 -> be59b68d5ff)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/doris-website.git discard 8ff698807d1 Automated deployment with doris branch @ 5cf3d2cf00e7c30dcc6cc180db816c89e753c694 new be59b68d5ff Automated deployment with doris branch @ 5cf3d2cf00e7c30dcc6cc180db816c89e753c694 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (8ff698807d1) \ N -- N -- N refs/heads/asf-site (be59b68d5ff) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/1.2/search-index.json | 2 +- docs/2.0/search-index.json | 2 +- docs/3.0/search-index.json | 2 +- docs/dev/search-index.json | 2 +- search-index.json| 2 +- zh-CN/docs/1.2/search-index.json | 2 +- zh-CN/docs/2.0/search-index.json | 2 +- zh-CN/docs/3.0/search-index.json | 2 +- zh-CN/docs/dev/search-index.json | 2 +- zh-CN/search-index.json | 2 +- 10 files changed, 10 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] Support move truncated old data to recycle bin [doris]
doris-robot commented on PR #43107: URL: https://github.com/apache/doris/pull/43107#issuecomment-2452876454 TPC-H: Total hot run time: 41504 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit cd197f854edbb4df9759080f0db57e11601695d3, data reload: false -- Round 1 -- q1 17583 738573137313 q2 2043162 155 155 q3 10577 110311951103 q4 10816 844 770 770 q5 7755301430813014 q6 243 147 149 147 q7 1043623 608 608 q8 9384202420332024 q9 6804648265506482 q10 7034246824782468 q11 469 262 258 258 q12 411 216 203 203 q13 17833 302830363028 q14 247 211 209 209 q15 572 507 506 506 q16 661 587 575 575 q17 970 531 538 531 q18 7373677967646764 q19 1331961 957 957 q20 447 188 183 183 q21 4228314131853141 q22 1162106510781065 Total cold run time: 108986 ms Total hot run time: 41504 ms - Round 2, with runtime_filter_mode=off - q1 7385748974047404 q2 336 243 243 243 q3 3293313431513134 q4 2193190719121907 q5 5902593058815881 q6 232 154 147 147 q7 2267179717991797 q8 3392366535733573 q9 8836892888458845 q10 3590353835723538 q11 623 498 525 498 q12 825 636 644 636 q13 14044 320032393200 q14 300 272 281 272 q15 570 520 522 520 q16 697 642 640 640 q17 1856164715971597 q18 8440790075247524 q19 1749164214631463 q20 2134189118331833 q21 5703547054585458 q22 1217106010311031 Total cold run time: 75584 ms Total hot run time: 61141 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Enhancement] (nereids)implement showPluginsCommand in nereids [doris]
Vallishp opened a new pull request, #43131:
URL: https://github.com/apache/doris/pull/43131
### What problem does this PR solve?
implement show plugins in nereids
Issue Number: close #42733
Related PR: #xxx
Problem Summary:
### Check List (For Committer)
- Test
- [ ] Regression test already exist.
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No colde files have been changed.
- [ ] Other reason
mysql> show plugins\G
*** 1. row ***
Name: __builtin_AuditLoader
Type: AUDIT
Description: builtin audit loader, to load audit log to internal table
Version: 2.1.0
JavaVersion: 1.8.31
ClassName: org.apache.doris.plugin.audit.AuditLoader
SoName: NULL
Sources: Builtin
Status: INSTALLED
Properties: {}
*** 2. row ***
Name: __builtin_AuditLogBuilder
Type: AUDIT
Description: builtin audit logger
Version: 0.12.0
JavaVersion: 1.8.31
ClassName: org.apache.doris.plugin.audit.AuditLogBuilder
SoName: NULL
Sources: Builtin
Status: INSTALLED
Properties: {}
*** 3. row ***
Name: __builtin_SqlDialectConverter
Type: DIALECT
Description: builtin sql dialect converter
Version: 2.1.0
JavaVersion: 1.8.31
ClassName: org.apache.doris.plugin.dialect.HttpDialectConverterPlugin
SoName: NULL
Sources: Builtin
Status: INSTALLED
Properties: {}
3 rows in set (3.17 sec)
- Behavior changed:
- [ ] No.
- [ ] Yes.
- Does this need documentation?
- [ ] No.
- [ ] Yes.
- Release note
None
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] (nereids)implement showPluginsCommand in nereids [doris]
doris-robot commented on PR #43131: URL: https://github.com/apache/doris/pull/43131#issuecomment-2452878325 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] (nereids)implement showPluginsCommand in nereids [doris]
Vallishp commented on PR #43131: URL: https://github.com/apache/doris/pull/43131#issuecomment-2452878399 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](type cast) Fix some implicit cast [doris]
doris-robot commented on PR #43050: URL: https://github.com/apache/doris/pull/43050#issuecomment-2452879028 TeamCity be ut coverage result: Function Coverage: 37.81% (9819/25967) Line Coverage: 28.98% (81611/281578) Region Coverage: 28.25% (42128/149152) Branch Coverage: 24.82% (21374/86106) Coverage Report: http://coverage.selectdb-in.cc/coverage/20be302463216192e44eb1dc10caabded1dcdfe6_20be302463216192e44eb1dc10caabded1dcdfe6/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Refactor](pipeline) Remove some unused code in pipeline query engine [doris]
HappenLee merged PR #42929: URL: https://github.com/apache/doris/pull/42929 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Refactor](pipeline) Remove some unused code in pipeline query engine (#42929)
This is an automated email from the ASF dual-hosted git repository.
lihaopeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new bb0c62696f0 [Refactor](pipeline) Remove some unused code in pipeline
query engine (#42929)
bb0c62696f0 is described below
commit bb0c62696f0fc7b150af1cc500d96f6de01dc8ce
Author: HappenLee
AuthorDate: Fri Nov 1 19:58:28 2024 +0800
[Refactor](pipeline) Remove some unused code in pipeline query engine
(#42929)
1. change some api unify
2. remove some unused code
---
be/src/pipeline/dependency.cpp | 15 +++
be/src/pipeline/dependency.h | 12
be/src/pipeline/pipeline.h | 7 ---
be/src/pipeline/pipeline_fragment_context.cpp| 1 -
be/src/pipeline/task_scheduler.h | 7 ++-
be/src/runtime/exec_env_init.cpp | 2 +-
be/src/runtime/workload_group/workload_group.cpp | 4 ++--
7 files changed, 16 insertions(+), 32 deletions(-)
diff --git a/be/src/pipeline/dependency.cpp b/be/src/pipeline/dependency.cpp
index 8d82c340e2d..5fef018423d 100644
--- a/be/src/pipeline/dependency.cpp
+++ b/be/src/pipeline/dependency.cpp
@@ -34,13 +34,14 @@
namespace doris::pipeline {
#include "common/compile_check_begin.h"
Dependency* BasicSharedState::create_source_dependency(int operator_id, int
node_id,
- std::string name) {
+ const std::string&
name) {
source_deps.push_back(std::make_shared(operator_id, node_id,
name + "_DEPENDENCY"));
source_deps.back()->set_shared_state(this);
return source_deps.back().get();
}
-Dependency* BasicSharedState::create_sink_dependency(int dest_id, int node_id,
std::string name) {
+Dependency* BasicSharedState::create_sink_dependency(int dest_id, int node_id,
+ const std::string& name) {
sink_deps.push_back(std::make_shared(dest_id, node_id, name +
"_DEPENDENCY", true));
sink_deps.back()->set_shared_state(this);
return sink_deps.back().get();
@@ -105,16 +106,6 @@ std::string RuntimeFilterDependency::debug_string(int
indentation_level) {
return fmt::to_string(debug_string_buffer);
}
-Dependency* RuntimeFilterDependency::is_blocked_by(PipelineTask* task) {
-std::unique_lock lc(_task_lock);
-auto ready = _ready.load();
-if (!ready && task) {
-_add_block_task(task);
-task->_blocked_dep = this;
-}
-return ready ? nullptr : this;
-}
-
void RuntimeFilterTimer::call_timeout() {
_parent->set_ready();
}
diff --git a/be/src/pipeline/dependency.h b/be/src/pipeline/dependency.h
index c6100fe0d8b..4cc3aceaeeb 100644
--- a/be/src/pipeline/dependency.h
+++ b/be/src/pipeline/dependency.h
@@ -17,6 +17,7 @@
#pragma once
+#include
#include
#include
@@ -27,7 +28,6 @@
#include
#include "common/logging.h"
-#include "concurrentqueue.h"
#include "gutil/integral_types.h"
#include "pipeline/common/agg_utils.h"
#include "pipeline/common/join_utils.h"
@@ -81,17 +81,15 @@ struct BasicSharedState {
virtual ~BasicSharedState() = default;
-Dependency* create_source_dependency(int operator_id, int node_id,
std::string name);
+Dependency* create_source_dependency(int operator_id, int node_id, const
std::string& name);
-Dependency* create_sink_dependency(int dest_id, int node_id, std::string
name);
+Dependency* create_sink_dependency(int dest_id, int node_id, const
std::string& name);
};
class Dependency : public std::enable_shared_from_this {
public:
ENABLE_FACTORY_CREATOR(Dependency);
-Dependency(int id, int node_id, std::string name)
-: _id(id), _node_id(node_id), _name(std::move(name)),
_ready(false) {}
-Dependency(int id, int node_id, std::string name, bool ready)
+Dependency(int id, int node_id, std::string name, bool ready = false)
: _id(id), _node_id(node_id), _name(std::move(name)),
_ready(ready) {}
virtual ~Dependency() = default;
@@ -278,8 +276,6 @@ public:
: Dependency(id, node_id, name), _runtime_filter(runtime_filter) {}
std::string debug_string(int indentation_level = 0) override;
-Dependency* is_blocked_by(PipelineTask* task) override;
-
private:
const IRuntimeFilter* _runtime_filter = nullptr;
};
diff --git a/be/src/pipeline/pipeline.h b/be/src/pipeline/pipeline.h
index 98e52ec5271..b969186b178 100644
--- a/be/src/pipeline/pipeline.h
+++ b/be/src/pipeline/pipeline.h
@@ -44,8 +44,7 @@ class Pipeline : public
std::enable_shared_from_this {
friend class PipelineFragmentContext;
public:
-explicit Pipeline(PipelineId pipeline_id, int num_tasks,
- std::weak_ptr context, int
num_tasks_of_parent)
+explicit Pipeli
Re: [PR] [test](mv) Fix regression test out text is wrong [doris]
seawinde commented on PR #43095: URL: https://github.com/apache/doris/pull/43095#issuecomment-2451760168 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](profile) update_rpc_time when enable_verbose_profile = false [doris]
Mryange commented on PR #43096: URL: https://github.com/apache/doris/pull/43096#issuecomment-2451761077 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](statistics)External table getRowCount return -1 when row count is not available or row count is 0. (#43009) [doris]
doris-robot commented on PR #43087: URL: https://github.com/apache/doris/pull/43087#issuecomment-2451759738 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 27f72a49ade7ba09ed2abea13049e3b2c4bdf412 with default session variables Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 21.5 seconds inserted 1000 Rows, about 465K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](profile) update_rpc_time when enable_verbose_profile = false [doris]
doris-robot commented on PR #43096: URL: https://github.com/apache/doris/pull/43096#issuecomment-2451760612 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](profile) update_rpc_time when enable_verbose_profile = false [doris]
Mryange opened a new pull request, #43096: URL: https://github.com/apache/doris/pull/43096 ### What problem does this PR solve? If enable_verbose_profile is false, update_rpc_time will not be called, and RPC count, max, and min statistics will not be recorded. ``` - RpcCount: 0 - RpcMaxTime: 0ns - RpcMinTime: 0ns - RpcSumTime: 0ns ``` Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Check List (For Committer) - Test - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No colde files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [ ] Yes. - Does this need documentation? - [ ] No. - [ ] Yes. - Release note None ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](statistics)External table getRowCount return -1 when row count is not available or row count is 0. (#43009) [doris]
doris-robot commented on PR #43087: URL: https://github.com/apache/doris/pull/43087#issuecomment-2451756148 ClickBench: Total hot run time: 30.37 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 27f72a49ade7ba09ed2abea13049e3b2c4bdf412, data reload: false query1 0.020.030.03 query2 0.070.030.03 query3 0.250.050.04 query4 1.780.100.07 query5 0.530.520.52 query6 1.270.610.60 query7 0.020.000.01 query8 0.040.020.02 query9 0.530.500.48 query10 0.540.530.54 query11 0.120.080.10 query12 0.120.090.09 query13 0.620.610.61 query14 0.780.780.79 query15 0.800.760.77 query16 0.390.370.35 query17 0.981.020.98 query18 0.210.280.24 query19 1.961.841.84 query20 0.010.020.01 query21 15.45 0.550.56 query22 2.102.561.49 query23 17.13 0.920.97 query24 5.783.370.92 query25 0.370.110.07 query26 0.820.160.15 query27 0.040.030.04 query28 5.340.760.72 query29 12.75 2.292.20 query30 0.620.530.55 query31 2.800.380.37 query32 3.380.490.49 query33 3.093.053.08 query34 15.29 4.784.78 query35 4.864.804.81 query36 1.061.021.01 query37 0.060.040.05 query38 0.040.020.02 query39 0.020.010.01 query40 0.160.140.13 query41 0.070.020.01 query42 0.020.020.01 query43 0.020.020.01 Total cold run time: 102.31 s Total hot run time: 30.37 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [test](mv) Fix regression test out text is wrong [doris]
seawinde opened a new pull request, #43095: URL: https://github.com/apache/doris/pull/43095 ### What problem does this PR solve? When shrinking the test data set, the corresponding out file data was not refreshed. As a result, the data in the out file was refreshed again. There are some tests that hit both materialized views as expected, modifying such corresponding cases. Issue Number: close #xxx Related PR: #40860 #41644 Problem Summary: ### Check List (For Committer) - Test - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No colde files have been changed. - [ ] Other reason - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. - Release note None ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](mv) Fix regression test out text is wrong [doris]
doris-robot commented on PR #43095: URL: https://github.com/apache/doris/pull/43095#issuecomment-2451757305 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](store) allow to set storage_page_size for tables [doris]
github-actions[bot] commented on PR #42680: URL: https://github.com/apache/doris/pull/42680#issuecomment-2451757385 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](type cast) Fix some implicit cast [doris]
zhiqiang- commented on PR #43050: URL: https://github.com/apache/doris/pull/43050#issuecomment-2451764171 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](mv) Fix regression test out text is wrong [doris]
github-actions[bot] commented on PR #43095: URL: https://github.com/apache/doris/pull/43095#issuecomment-2451768442 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](mv) Fix regression test out text is wrong [doris]
github-actions[bot] commented on PR #43095: URL: https://github.com/apache/doris/pull/43095#issuecomment-2451768493 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](type cast) Fix some implicit cast [doris]
github-actions[bot] commented on PR #43050: URL: https://github.com/apache/doris/pull/43050#issuecomment-2451770669 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](Nereids) boost characterLiteralTypeCoercion (#42941) [doris]
doris-robot commented on PR #43092: URL: https://github.com/apache/doris/pull/43092#issuecomment-2451727733 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](auditlog) add missing audit log fields and duplicate audit log error [doris]
morningman commented on PR #42262: URL: https://github.com/apache/doris/pull/42262#issuecomment-2451852328 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Export) fix dead lock of parallel Exporting [doris]
doris-robot commented on PR #43097: URL: https://github.com/apache/doris/pull/43097#issuecomment-2451840895 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 7a04d53352df5ffc455ae9fec0f827ebd500c020 with default session variables Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s Insert into select: 21.4 seconds inserted 1000 Rows, about 467K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](auto-pick)Support cherry pick to 2.1 [doris]
github-actions[bot] commented on PR #43106: URL: https://github.com/apache/doris/pull/43106#issuecomment-2451875164 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] branch-3.0: [fix](jdbc catalog) Optimize JDBC Connection Closing to Ensure ProperResource Release [doris]
github-actions[bot] opened a new pull request, #43105: URL: https://github.com/apache/doris/pull/43105 PR Body: This pull request optimizes the JDBC connection and resource management in `JdbcClient` and related methods. The main changes are as follows: - **Manual Resource Closing**: Explicitly place resource acquisition and closing within a `try-catch-finally` structure. In the `finally` block, manually close resources like `Connection`, `Statement`, and `ResultSet` to ensure that resources are properly released even if an exception occurs. - **Remove try-with-resources**: All usages of `try-with-resources` have been removed, switching to manual resource management instead. - **Improve Exception Handling**: When closing resources, if an exception occurs, log a warning but do not throw a new exception. This avoids masking the original exception during the resource release phase. - **Unified Resource Management**: Ensure consistent resource acquisition and closing methods across all functions, improving code readability and maintainability. Functions modified include but are not limited to: - `executeStmt` - `getColumnsFromQuery` - `getDatabaseNameList` - `getJdbcColumnsInfo` - `processTable` - `getColumnsDataTypeUseQuery` These changes enhance the robustness of the code, prevent potential resource leaks, and ensure that all JDBC connections and resources are properly closed without using `try-with-resources`. Cherry-picked from #43059 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](auto-pick)Support cherry pick to 2.1 [doris]
github-actions[bot] commented on PR #43106: URL: https://github.com/apache/doris/pull/43106#issuecomment-2451875105 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](cloud) refine block file cache evict policy [doris]
doris-robot commented on PR #42451: URL: https://github.com/apache/doris/pull/42451#issuecomment-2451854953 TPC-H: Total hot run time: 41512 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b8e12301562faf64063402ef95381cda46903ae0, data reload: false -- Round 1 -- q1 17577 768473437343 q2 2052174 173 173 q3 10595 116712361167 q4 10223 829 876 829 q5 7753328030453045 q6 238 149 148 148 q7 1004610 598 598 q8 9364203121112031 q9 6620647464536453 q10 7066246324222422 q11 459 247 256 247 q12 407 212 211 211 q13 17798 300530333005 q14 253 221 217 217 q15 576 519 522 519 q16 653 583 581 581 q17 1006582 499 499 q18 7322671167086708 q19 13231057941 941 q20 484 175 185 175 q21 4046325032093209 q22 1112991 1012991 Total cold run time: 107931 ms Total hot run time: 41512 ms - Round 2, with runtime_filter_mode=off - q1 7352726772777267 q2 333 230 224 224 q3 3093305530703055 q4 2002179717951795 q5 5748570857535708 q6 233 141 143 141 q7 2251178118001781 q8 3378363035373537 q9 9014902690249024 q10 3645360035573557 q11 614 496 506 496 q12 816 622 625 622 q13 8343322830473047 q14 298 275 264 264 q15 600 512 503 503 q16 695 616 620 616 q17 1898166016261626 q18 8159766378407663 q19 1748167215981598 q20 2158188318961883 q21 5399551053985398 q22 1174104210141014 Total cold run time: 68951 ms Total hot run time: 60819 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (eaaf5b4b373 -> a8ecb19cb50)
This is an automated email from the ASF dual-hosted git repository. kirs pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from eaaf5b4b373 [fix](jdbc catalog) Optimize JDBC Connection Closing to Ensure ProperResource Release (#43059) add a8ecb19cb50 [chore](auto-pick)Support cherry pick to 2.1 (#43106) No new revisions were added by this update. Summary of changes: .github/workflows/auto-cherry-pick.yml | 13 +++-- 1 file changed, 11 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](auto-pick)Support cherry pick to 2.1 [doris]
CalvinKirs merged PR #43106: URL: https://github.com/apache/doris/pull/43106 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](persist) Fix the generated dropping tmp partition sql [doris]
dataroaring commented on PR #43079: URL: https://github.com/apache/doris/pull/43079#issuecomment-2451883127 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [doc] (fluentbit) fluentbit doris output plugin [doris-website]
KassieZ commented on code in PR #1260:
URL: https://github.com/apache/doris-website/pull/1260#discussion_r1825896425
##
common_docs_zh/ecosystem/fluentbit.md:
##
@@ -0,0 +1,369 @@
+---
+{
+"title": "Fluent Bit Doris Output Plugin",
+"language": "zh-CN"
+}
+---
+
+
+
+# Fluent Bit Doris output plugin
+
+[Fluent Bit](https://fluentbit.io/) 是一个快速的日志处理器和转发器，它支持自定义输出插件将数据写入存储系统，Fluent
Bit Doris output plugin 是输出到 Doris 的插件。
+
+Fluent Bit Doris output plugin 调用 [Doris Stream
Load](../data-operate/import/stream-load-manual.md) HTTP 接口将数据实时写入
Doris，提供多线程并发，失败重试，自定义 Stream Load 格式和参数，输出写入速度等能力。
+
+使用 Fluent Bit Doris output plugin 主要有三个步骤：
+1. 下载或编译包含 Doris output plugin 的 Fluent Bit 二进制程序
+2. 配置 Fluent Bit 输出地址和其他参数
+3. 启动 Fluent Bit 将数据实时写入 Doris
+
+## 安装（alpha）
+
+### 从官网下载
+
+https://github.com/joker-star-l/fluent-bit/releases/download/v3.1.9-alpha/fluent-bit
+
+### 从源码编译
+
+克隆 https://github.com/joker-star-l/fluent-bit 的 dev 分支，在 build/ 目录下执行
+
+```
+cmake -DFLB_RELEASE=ON ..
+make
+```
+
+编译产物为 build/bin/fluent-bit。
+
+## 参数配置
+
+Fluent Bit Doris output plugin 的配置如下：
+
+配置 | 说明
+--- | ---
+`host` | Stream Load HTTP host
+`port` | Stream Load HTTP port
+`user` | Doris 用户名，该用户需要有doris对应库表的导入权限
+`password` | Doris 用户的密码
+`database` | 要写入的 Doris 库名
+`table` | 要写入的 Doris 表名
+`label_prefix` | Doris Stream Load Label 前缀，最终生成的 Label 为
*{label_prefix}\_{timestamp}\_{uuid}* ，默认值是 fluentbit, 如果设置为 false 则不会添加 Label
+ `time_key` | 数据中要添加的时间戳列的名称，默认值是 date， 如果设置为 false 则不会添加该列
+`header` | Doris Stream Load 的 header 参数，可以设置多个
+`log_request` | 日志中是否输出 Doris Stream Load 请求和响应元数据，用于排查问题，默认为 true
+`log_progress_interval` | 日志中输出速度的时间间隔，单位是秒，默认为 10，设置为 0 可以关闭这种日志
+`retry_limit` | Doris Stream Load 请求失败重试次数，默认为 1, 如果设置为 false 则不限制重试次数
+`workers` | 执行 Doris Stream Load 的 worker 数量，默认为 2
+
+## 使用示例
+
+### TEXT 日志采集示例
+
+该示例以 Doris FE 的日志为例展示 TEXT 日志采集。
+
+**1. 数据**
+
+FE 日志文件一般位于 Doris 安装目录下的 fe/log/fe.log 文件，是典型的 Java
程序日志，包括时间戳，日志级别，线程名，代码位置，日志内容等字段。不仅有正常的日志，还有带 stacktrace 的异常日志，stacktrace
是跨行的，日志采集存储需要把主日志和 stacktrace 组合成一条日志。
+
+```
+2024-07-08 21:18:01,432 INFO (Statistics Job Appender|61)
[StatisticsJobAppender.runAfterCatalogReady():70] Stats table not available,
skip
+2024-07-08 21:18:53,710 WARN (STATS_FETCH-0|208)
[StmtExecutor.executeInternalQuery():3332] Failed to run internal SQL:
OriginStatement{originStmt='SELECT * FROM __internal_schema.column_statistics
WHERE part_id is NULL ORDER BY update_time DESC LIMIT 50', idx=0}
+org.apache.doris.common.UserException: errCode = 2, detailMessage = tablet
10031 has no queryable replicas. err: replica 10032's backend 10008 does not
exist or not alive
+at
org.apache.doris.planner.OlapScanNode.addScanRangeLocations(OlapScanNode.java:931)
~[doris-fe.jar:1.2-SNAPSHOT]
+at
org.apache.doris.planner.OlapScanNode.computeTabletInfo(OlapScanNode.java:1197)
~[doris-fe.jar:1.2-SNAPSHOT]
+```
+
+**2. 建表**
+
+表结构包括日志的产生时间，采集时间，主机名，日志文件路径，日志类型，日志级别，线程名，代码位置，日志内容等字段。
+
+```
+CREATE TABLE `doris_log` (
+ `log_time` datetime NULL COMMENT 'log content time',
+ `collect_time` datetime NULL COMMENT 'log agent collect time',
+ `host` text NULL COMMENT 'hostname or ip',
+ `path` text NULL COMMENT 'log file path',
+ `type` text NULL COMMENT 'log type',
+ `level` text NULL COMMENT 'log level',
+ `thread` text NULL COMMENT 'log thread',
+ `position` text NULL COMMENT 'log code position',
+ `message` text NULL COMMENT 'log message',
+ INDEX idx_host (`host`) USING INVERTED COMMENT '',
+ INDEX idx_path (`path`) USING INVERTED COMMENT '',
+ INDEX idx_type (`type`) USING INVERTED COMMENT '',
+ INDEX idx_level (`level`) USING INVERTED COMMENT '',
+ INDEX idx_thread (`thread`) USING INVERTED COMMENT '',
+ INDEX idx_position (`position`) USING INVERTED COMMENT '',
+ INDEX idx_message (`message`) USING INVERTED PROPERTIES("parser" =
"unicode", "support_phrase" = "true") COMMENT ''
+) ENGINE=OLAP
+DUPLICATE KEY(`log_time`)
+COMMENT 'OLAP'
+PARTITION BY RANGE(`log_time`) ()
+DISTRIBUTED BY RANDOM BUCKETS 10
+PROPERTIES (
+"replication_num" = "1",
+"dynamic_partition.enable" = "true",
+"dynamic_partition.time_unit" = "DAY",
+"dynamic_partition.start" = "-7",
+"dynamic_partition.end" = "1",
+"dynamic_partition.prefix" = "p",
+"dynamic_partition.buckets" = "10",
+"dynamic_partition.create_history_partition" = "true",
+"compaction_policy" = "time_series"
+);
+```
+
+**3. 配置**
+
+Fluent Bit 日志采集的配置文件如下，doris_log.conf 用于定义 ETL 的各个部分组件，parsers.conf
用于定义不同的日志解析器。
+
+doris_log.conf:
+```
+# config for Fluent Bit service
+[SERVICE]
+log_level info
+# parsers file
+parsers_file parsers.conf
+
+# use input tail
+[INPUT]
+name tail
+path /path/to/your/log
+# add log file name to the record, key is 'path'
+path_key path
+# set multiline parser
+multiline.parser multiline_java
+
+# parse log
+[FILTER]
+match *
+name parser
+key_name log
+parser fe_log
+reserve_data true
+
+# add host info
+[FILTER
Re: [PR] [fix](export) fix potential export concurrency issue [doris]
doris-robot commented on PR #43109: URL: https://github.com/apache/doris/pull/43109#issuecomment-2451983156 TPC-DS: Total hot run time: 196008 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 9c84473091fc70d1f67d3501e21a7f4d37dc302f, data reload: false query1 1213933 897 897 query2 6234219821272127 query3 10806 413540084008 query4 67901 27665 23699 23699 query5 4881444 435 435 query6 398 183 165 165 query7 5591305 296 296 query8 313 226 227 226 query9 8947272427232723 query10 463 253 249 249 query11 17359 15271 15998 15271 query12 157 98 107 98 query13 1567425 417 417 query14 10322 718076137180 query15 210 182 189 182 query16 7143483 489 483 query17 1042548 557 548 query18 1764306 294 294 query19 191 159 145 145 query20 118 112 112 112 query21 196 98 100 98 query22 4663429144554291 query23 34508 33945 34440 33945 query24 6011278027852780 query25 519 393 405 393 query26 654 156 157 156 query27 1670278 284 278 query28 4116242924082408 query29 683 429 415 415 query30 230 157 155 155 query31 986 783 815 783 query32 62 56 58 56 query33 401 260 269 260 query34 894 514 501 501 query35 860 741 751 741 query36 1085961 958 958 query37 119 79 70 70 query38 4363421542604215 query39 1512144414151415 query40 203 99 101 99 query41 47 45 45 45 query42 110 101 98 98 query43 550 506 512 506 query44 1158814 812 812 query45 185 166 172 166 query46 1108699 699 699 query47 1954185418421842 query48 426 313 321 313 query49 748 391 415 391 query50 806 395 391 391 query51 7341711171157111 query52 98 87 83 83 query53 247 176 183 176 query54 505 387 383 383 query55 77 78 74 74 query56 248 225 223 223 query57 1277117411471147 query58 217 201 202 201 query59 3378307529672967 query60 274 239 234 234 query61 104 97 100 97 query62 784 677 677 677 query63 212 186 177 177 query64 1327654 610 610 query65 3295321233053212 query66 695 306 301 301 query67 15987 15619 15765 15619 query68 3756561 563 561 query69 416 251 254 251 query70 1171115810771077 query71 352 251 248 248 query72 6232401140444011 query73 755 350 358 350 query74 10027 924790269026 query75 3396267326502650 query76 1809107810371037 query77 488 278 262 262 query78 10401 939393079307 query79 2167581 586 581 query80 981 413 413 413 query81 538 239 245 239 query82 212 120 121 120 query83 159 136 133 133 query84 286 70 68 68 query85 988 295 293 293 query86 421 309 277 277 query87 4850473546974697 query88 4358219221582158 query89 406 293 285 285 query90 2017201 188 188 query91 137 99 103 99 query92 64 48 50 48 query93 2305535 536 535 query94 932 292 293 292 query95 354 264 247 247 query96 620 282 284 282 query97 2872272626642664 query98 212 203 196 196 query99 1587133012861286 Total cold run time: 318782 ms Total hot run time: 196008 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [fix](cloud-mow) MS should delete the existing keys before rewriting it when processing old version delete bitmap on cu compaction [doris]
doris-robot commented on PR #42379: URL: https://github.com/apache/doris/pull/42379#issuecomment-2451984908 ClickBench: Total hot run time: 32.66 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 74f1a52095d6f08bfa92c796446509805e467fa0, data reload: false query1 0.030.030.03 query2 0.060.030.03 query3 0.230.060.06 query4 1.640.100.09 query5 0.430.410.39 query6 1.160.650.66 query7 0.020.020.01 query8 0.040.030.03 query9 0.570.490.49 query10 0.560.550.55 query11 0.140.100.12 query12 0.140.110.11 query13 0.610.600.60 query14 2.712.822.71 query15 0.920.830.82 query16 0.380.370.38 query17 1.041.021.07 query18 0.200.190.20 query19 1.991.841.96 query20 0.020.010.01 query21 15.35 0.590.59 query22 2.511.872.08 query23 16.95 1.120.95 query24 2.821.560.99 query25 0.260.040.06 query26 0.530.140.14 query27 0.040.040.04 query28 10.54 1.101.06 query29 12.54 3.223.20 query30 0.240.060.06 query31 2.870.380.38 query32 3.290.460.46 query33 2.953.012.97 query34 16.89 4.494.52 query35 4.574.554.57 query36 0.660.470.49 query37 0.090.060.06 query38 0.050.040.03 query39 0.030.030.02 query40 0.160.120.12 query41 0.070.020.02 query42 0.040.020.02 query43 0.030.030.03 Total cold run time: 106.37 s Total hot run time: 32.66 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feat](Nereids) support unset command [doris]
doris-robot commented on PR #43103: URL: https://github.com/apache/doris/pull/43103#issuecomment-2451984968 ClickBench: Total hot run time: 32.04 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit a1babf84fde311041e691e61920b99eeaea6, data reload: false query1 0.040.040.03 query2 0.060.030.03 query3 0.220.060.06 query4 1.650.100.09 query5 0.410.400.41 query6 1.160.660.64 query7 0.010.020.02 query8 0.040.030.03 query9 0.560.500.49 query10 0.560.550.57 query11 0.140.100.11 query12 0.140.110.12 query13 0.610.590.59 query14 2.842.732.72 query15 0.900.820.83 query16 0.390.370.38 query17 1.041.041.04 query18 0.200.200.20 query19 1.831.781.93 query20 0.020.010.01 query21 15.37 0.590.57 query22 2.762.091.45 query23 17.11 0.910.86 query24 2.741.140.76 query25 0.150.180.08 query26 0.550.150.14 query27 0.040.040.04 query28 10.92 1.101.07 query29 12.54 3.243.24 query30 0.250.060.05 query31 2.890.390.37 query32 3.270.470.46 query33 3.003.043.02 query34 16.84 4.524.50 query35 4.574.584.58 query36 0.660.480.48 query37 0.090.060.06 query38 0.050.030.03 query39 0.030.020.03 query40 0.160.120.11 query41 0.070.020.02 query42 0.030.020.02 query43 0.040.030.03 Total cold run time: 106.95 s Total hot run time: 32.04 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](segment compaction) fix error using of inverted index file writer in segment compaction [doris]
doris-robot commented on PR #43114: URL: https://github.com/apache/doris/pull/43114#issuecomment-2451984312 TeamCity be ut coverage result: Function Coverage: 37.82% (9820/25968) Line Coverage: 28.98% (81607/281571) Region Coverage: 28.25% (42135/149156) Branch Coverage: 24.83% (21379/86108) Coverage Report: http://coverage.selectdb-in.cc/coverage/d0062bc0e9991e30b1b6a5630871edf9c17c4ac3_d0062bc0e9991e30b1b6a5630871edf9c17c4ac3/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] Support move truncated old data to recycle bin [doris]
doris-robot commented on PR #43107: URL: https://github.com/apache/doris/pull/43107#issuecomment-2451987317 ClickBench: Total hot run time: 32.42 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 0b29c92f47071d6053cc4508e8117802bedb36fa, data reload: false query1 0.040.030.02 query2 0.070.040.03 query3 0.230.060.06 query4 1.640.100.10 query5 0.420.400.41 query6 1.140.660.64 query7 0.020.010.02 query8 0.040.030.03 query9 0.570.500.49 query10 0.550.560.55 query11 0.140.100.10 query12 0.140.110.13 query13 0.600.600.59 query14 2.832.742.82 query15 0.900.830.83 query16 0.400.380.39 query17 0.981.051.05 query18 0.210.190.20 query19 1.961.811.94 query20 0.020.000.01 query21 15.36 0.570.58 query22 2.622.441.70 query23 17.04 1.020.83 query24 2.941.091.24 query25 0.300.040.23 query26 0.450.140.14 query27 0.040.050.04 query28 10.59 1.101.08 query29 12.61 3.263.24 query30 0.250.060.06 query31 2.850.390.38 query32 3.260.460.45 query33 2.962.983.03 query34 17.13 4.444.52 query35 4.514.564.48 query36 0.670.470.48 query37 0.090.060.06 query38 0.040.030.04 query39 0.040.020.02 query40 0.160.130.12 query41 0.070.020.02 query42 0.030.020.03 query43 0.040.030.03 Total cold run time: 106.95 s Total hot run time: 32.42 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Enhancement] (nereids)implement showBackendsCommand in nereids [doris]
Vallishp opened a new pull request, #43121:
URL: https://github.com/apache/doris/pull/43121
### What problem does this PR solve?
Support Show Backends in nereids
Issue Number: close #42762
Related PR: #xxx
Problem Summary:
### Check List (For Committer)
- Test
- [ ] Regression test yes
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No colde files have been changed.
- [ ] Other reason
Already testcase present test_backend.groovy
RemoteUsedCapacity:0.000 , BackendId:10002, CpuCores:8,
LastStartTime:2024-11-01 17:45:25, Tag:{"location" : "default"}]]
2024-11-01 17:49:17.647 INFO [non-concurrent-thread-1] (Suite.groovy:384) -
Execute sql: ALTER SYSTEM DECOMMISSION BACKEND "10002"
2024-11-01 17:49:17.666 INFO [non-concurrent-thread-1]
(test_backend.groovy:60) - ALTER SYSTEM DECOMMISSION BACKEND [[0]]
2024-11-01 17:49:17.667 INFO [non-concurrent-thread-1] (Suite.groovy:451) -
Execute sql: SHOW BACKENDS;
2024-11-01 17:49:17.694 INFO [non-concurrent-thread-1] (Suite.groovy:384) -
Execute sql: CANCEL DECOMMISSION BACKEND "10002"
2024-11-01 17:49:17.707 INFO [non-concurrent-thread-1]
(test_backend.groovy:71) - CANCEL DECOMMISSION BACKEND [[0]]
2024-11-01 17:49:17.708 INFO [non-concurrent-thread-1] (Suite.groovy:451) -
Execute sql: SHOW BACKENDS;
2024-11-01 17:49:17.718 INFO [non-concurrent-thread-1] (Suite.groovy:451) -
Execute sql: show frontends
2024-11-01 17:49:17.805 INFO [non-concurrent-thread-1]
(DebugPoint.groovy:162) - disable debug point
SystemHandler.decommission_no_check_replica_num for FE 172.20.80.1:8030
2024-11-01 17:49:17.819 INFO [non-concurrent-thread-1] (Http.groovy:85) -
http post
url=http://172.20.80.1:8030/api/debug_point/remove/SystemHandler.decommission_no_check_replica_num,
data=null, isJson=true, response code=200,
text={"msg":"success","code":0,"data":null,"count":0}
2024-11-01 17:49:17.820 INFO [non-concurrent-thread-1] (Suite.groovy:384) -
Execute sql: admin set frontend config("drop_backend_after_decommission" =
"true")
2024-11-01 17:49:17.825 INFO [non-concurrent-thread-1]
(ScriptContext.groovy:120) - Run test_backend in
/root/doris/workspace/doris/regression-test/suites/node_p0/test_backend.groovy
succeed
2024-11-01 17:49:25.546 INFO [main] (RegressionTest.groovy:410) - Success
suites:
/root/doris/workspace/doris/regression-test/suites/node_p0/test_backend.groovy:
group=nonConcurrent,p0, name=test_backend
2024-11-01 17:49:25.547 INFO [main] (RegressionTest.groovy:492) - All suites
success.
- Behavior changed:
- [ ] No.
- [ ] Yes.
- Does this need documentation?
- [ ] No.
- [ ] Yes.
- Release note
None
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] (nereids)implement showBackendsCommand in nereids [doris]
doris-robot commented on PR #43121: URL: https://github.com/apache/doris/pull/43121#issuecomment-2452319136 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] (nereids)implement showPrivilegesCommand in nereids [doris]
Vallishp commented on PR #43119: URL: https://github.com/apache/doris/pull/43119#issuecomment-2452320222 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] (nereids)implement showBackendsCommand in nereids [doris]
Vallishp commented on PR #43121: URL: https://github.com/apache/doris/pull/43121#issuecomment-2452319803 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression-test) fix test_hive_serde_prop #42886 [doris]
doris-robot commented on PR #43099: URL: https://github.com/apache/doris/pull/43099#issuecomment-2451788948 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression-test) fix test_hive_serde_prop #42886 [doris]
doris-robot commented on PR #43098: URL: https://github.com/apache/doris/pull/43098#issuecomment-2451788601 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] branch-3.0: [fix](regression-test) fix test_hive_serde_prop [doris]
github-actions[bot] opened a new pull request, #43100: URL: https://github.com/apache/doris/pull/43100 PR Body: ## Proposed changes fellow https://github.com/apache/doris/pull/42200 Null values in text tables are now correctly recognized now. Cherry-picked from #42886 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](regression-test) fix test_hive_serde_prop [doris]
doris-robot commented on PR #43100: URL: https://github.com/apache/doris/pull/43100#issuecomment-2451788952 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](regression-test) fix test_hive_serde_prop [doris]
github-actions[bot] commented on PR #43100: URL: https://github.com/apache/doris/pull/43100#issuecomment-2451788898 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression-test) fix test_hive_serde_prop #42886 [doris]
morningman commented on PR #43098: URL: https://github.com/apache/doris/pull/43098#issuecomment-2451788562 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-42886-branch-3.0 updated (db0986e512b -> 600666d0aae)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-42886-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from db0986e512b [fix](case)Fix multiple case tables with the same name causing case failure (#42203) (#43041) add 600666d0aae [fix](regression-test) fix test_hive_serde_prop (#42886) No new revisions were added by this update. Summary of changes: .../data/external_table_p0/hive/test_hive_serde_prop.out | 15 --- 1 file changed, 8 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] branch-3.0: [feature](doris compose) enable fe jvm debug port [doris]
github-actions[bot] opened a new pull request, #43108: URL: https://github.com/apache/doris/pull/43108 PR Body: 1. fe use 5005 as jvm debug port, user can run `docker ps | grep ` to find out the port on host machine; 2. extends get-cloud-ini command to print more info Cherry-picked from #43085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [feature](doris compose) enable fe jvm debug port [doris]
github-actions[bot] commented on PR #43108: URL: https://github.com/apache/doris/pull/43108#issuecomment-2451896224 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43085-branch-3.0 updated (594865c34f5 -> 1f423bf1a28)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43085-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from 594865c34f5 [improvement](statistics)External table getRowCount return -1 when row count is not available or row count is 0. (#43009) (#43081) add 1f423bf1a28 [feature](doris compose) enable fe jvm debug port (#43085) No new revisions were added by this update. Summary of changes: docker/runtime/doris-compose/cluster.py | 24 +-- docker/runtime/doris-compose/command.py | 76 +++-- 2 files changed, 56 insertions(+), 44 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [feature](doris compose) enable fe jvm debug port [doris]
doris-robot commented on PR #43108: URL: https://github.com/apache/doris/pull/43108#issuecomment-2451896324 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43085-branch-3.0 created (now 594865c34f5)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43085-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git at 594865c34f5 [improvement](statistics)External table getRowCount return -1 when row count is not available or row count is 0. (#43009) (#43081) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [fix](cloud) Fix constant folding for requests sent to BE without dis… [doris]
github-actions[bot] commented on PR #43110: URL: https://github.com/apache/doris/pull/43110#issuecomment-2451898845 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](case) fix mow cluster key case [doris]
github-actions[bot] commented on PR #43066: URL: https://github.com/apache/doris/pull/43066#issuecomment-2451897928 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-42865-branch-3.0 created (now 594865c34f5)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-42865-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git at 594865c34f5 [improvement](statistics)External table getRowCount return -1 when row count is not available or row count is 0. (#43009) (#43081) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](cloud) Fix constant folding for requests sent to BE without dis… (#42865)
This is an automated email from the ASF dual-hosted git repository.
kirs pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7e90e665f9b [fix](cloud) Fix constant folding for requests sent to BE
without dis… (#42865)
7e90e665f9b is described below
commit 7e90e665f9b331b30e2cbd2994e1f1086cf6a4bb
Author: deardeng <565620...@qq.com>
AuthorDate: Fri Nov 1 21:44:36 2024 +0800
[fix](cloud) Fix constant folding for requests sent to BE without dis…
(#42865)
…tinguishing clusters
## Proposed changes
Issue Number: close #xxx
replace function getAllBackendIds by getBackendsByCurrentCluster
---
.../rules/expression/rules/FoldConstantRuleOnBE.java | 2 +-
.../java/org/apache/doris/rewrite/FoldConstantsRule.java | 2 +-
.../org/apache/doris/service/FrontendServiceImpl.java | 4 ++--
.../java/org/apache/doris/system/SystemInfoService.java | 15 +--
.../doris/tablefunction/LocalTableValuedFunction.java | 2 +-
5 files changed, 18 insertions(+), 7 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnBE.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnBE.java
index dedd1b35bd7..64f7c2784f7 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnBE.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/FoldConstantRuleOnBE.java
@@ -270,7 +270,7 @@ public class FoldConstantRuleOnBE implements
ExpressionPatternRuleFactory {
Map constMap, ConnectContext context) {
Map resultMap = new HashMap<>();
try {
-List backendIds =
Env.getCurrentSystemInfo().getAllBackendIds(true);
+List backendIds =
Env.getCurrentSystemInfo().getAllBackendByCurrentCluster(true);
if (backendIds.isEmpty()) {
throw new UserException("No alive backends");
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/rewrite/FoldConstantsRule.java
b/fe/fe-core/src/main/java/org/apache/doris/rewrite/FoldConstantsRule.java
index d86d8c22564..6275cb9bb84 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/rewrite/FoldConstantsRule.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/rewrite/FoldConstantsRule.java
@@ -361,7 +361,7 @@ public class FoldConstantsRule implements ExprRewriteRule {
TNetworkAddress brpcAddress = null;
Map> resultMap = new HashMap<>();
try {
-List backendIds =
Env.getCurrentSystemInfo().getAllBackendIds(true);
+List backendIds =
Env.getCurrentSystemInfo().getAllBackendByCurrentCluster(true);
if (backendIds.isEmpty()) {
throw new LoadException("Failed to get all partitions. No
alive backends");
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
index cd1af82563d..e3d543d9111 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
@@ -3495,7 +3495,7 @@ public class FrontendServiceImpl implements
FrontendService.Iface {
// build nodes
List nodeInfos = Lists.newArrayList();
SystemInfoService systemInfoService = Env.getCurrentSystemInfo();
-for (Long id : systemInfoService.getAllBackendIds(false)) {
+for (Long id : systemInfoService.getAllBackendByCurrentCluster(false))
{
Backend backend = systemInfoService.getBackend(id);
nodeInfos.add(new TNodeInfo(backend.getId(), 0, backend.getHost(),
backend.getBrpcPort()));
}
@@ -3704,7 +3704,7 @@ public class FrontendServiceImpl implements
FrontendService.Iface {
// build nodes
List nodeInfos = Lists.newArrayList();
SystemInfoService systemInfoService = Env.getCurrentSystemInfo();
-for (Long id : systemInfoService.getAllBackendIds(false)) {
+for (Long id : systemInfoService.getAllBackendByCurrentCluster(false))
{
Backend backend = systemInfoService.getBackend(id);
nodeInfos.add(new TNodeInfo(backend.getId(), 0, backend.getHost(),
backend.getBrpcPort()));
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/system/SystemInfoService.java
b/fe/fe-core/src/main/java/org/apache/doris/system/SystemInfoService.java
index 69216dd4d13..447dc3457ae 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/system/SystemInfoService.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/system/SystemInfoService.java
@@ -164,7 +164,7 @@ public class SystemInfoService {
public static TPaloNodesInfo createAliveNodesInfo() {
T
Re: [PR] branch-3.0: [fix](cloud) Fix constant folding for requests sent to BE without dis… [doris]
doris-robot commented on PR #43110: URL: https://github.com/apache/doris/pull/43110#issuecomment-2451898940 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (7e90e665f9b -> c14b03c5bc7)
This is an automated email from the ASF dual-hosted git repository. kirs pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 7e90e665f9b [fix](cloud) Fix constant folding for requests sent to BE without dis… (#42865) add c14b03c5bc7 [test](stream load) add cases to ensure the correctness of the document (#43016) No new revisions were added by this update. Summary of changes: .../import/import-way/stream-load-manual.md.out| 154 +++ .../import/import-way/streamload_example.csv | 10 + .../import/import-way/streamload_example.json | 12 + .../data-operate/import/import-way/test_array.csv | 10 + .../data-operate/import/import-way/test_bitmap.csv | 10 + .../import/import-way/test_default.csv | 1 + .../import/import-way/test_enclose_and_escape.csv | 1 + .../data-operate/import/import-way/test_hll.csv| 10 + .../data-operate/import/import-way/test_map.json | 12 + .../import/import-way/test_merge_type.csv | 2 + .../data-operate/import/import-way/test_seq.csv| 1 + .../import/import-way/stream-load-manual.md.groovy | 492 + 12 files changed, 715 insertions(+) create mode 100644 regression-test/data/doc/data-operate/import/import-way/stream-load-manual.md.out create mode 100644 regression-test/data/doc/data-operate/import/import-way/streamload_example.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/streamload_example.json create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_array.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_bitmap.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_default.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_enclose_and_escape.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_hll.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_map.json create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_merge_type.csv create mode 100644 regression-test/data/doc/data-operate/import/import-way/test_seq.csv create mode 100644 regression-test/suites/doc/data-operate/import/import-way/stream-load-manual.md.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43085-branch-2.1 created (now 03a614f9452)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43085-branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git at 03a614f9452 [fix](Export) fix dead lock of parallel Exporting (#43084) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-43016-branch-3.0 created (now 594865c34f5)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-43016-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git at 594865c34f5 [improvement](statistics)External table getRowCount return -1 when row count is not available or row count is 0. (#43009) (#43081) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [test](stream load) add cases to ensure the correctness of the document [doris]
doris-robot commented on PR #43111: URL: https://github.com/apache/doris/pull/43111#issuecomment-2451900748 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] branch-3.0: [test](stream load) add cases to ensure the correctness of the document [doris]
github-actions[bot] opened a new pull request, #43111: URL: https://github.com/apache/doris/pull/43111 PR Body: Add cases to ensure the correctness of the document. Cherry-picked from #43016 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] branch-3.0: [test](stream load) add cases to ensure the correctness of the document [doris]
doris-robot commented on PR #43111: URL: https://github.com/apache/doris/pull/43111#issuecomment-2451900773 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch auto-pick-42865-branch-3.0 updated (594865c34f5 -> 6d82f9cef27)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch auto-pick-42865-branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from 594865c34f5 [improvement](statistics)External table getRowCount return -1 when row count is not available or row count is 0. (#43009) (#43081) add 6d82f9cef27 [fix](cloud) Fix constant folding for requests sent to BE without dis… (#42865) No new revisions were added by this update. Summary of changes: .../rules/expression/rules/FoldConstantRuleOnBE.java | 2 +- .../java/org/apache/doris/rewrite/FoldConstantsRule.java | 2 +- .../org/apache/doris/service/FrontendServiceImpl.java | 4 ++-- .../java/org/apache/doris/system/SystemInfoService.java | 15 +-- .../doris/tablefunction/LocalTableValuedFunction.java | 2 +- 5 files changed, 18 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](statistics)External table getRowCount return -1 when row count is not available or row count is 0. (#43009) [doris]
Jibing-Li commented on PR #43087: URL: https://github.com/apache/doris/pull/43087#issuecomment-2452012632 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression-test) fix test_hive_serde_prop #42886 [doris]
yiguolei merged PR #43098: URL: https://github.com/apache/doris/pull/43098 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](statistics)External table getRowCount return -1 when row count is not available or row count is 0. (#43009) [doris]
Jibing-Li commented on PR #43087: URL: https://github.com/apache/doris/pull/43087#issuecomment-2452055400 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Update local.md [doris-website]
lutinglt commented on PR #1242: URL: https://github.com/apache/doris-website/pull/1242#issuecomment-2452054510 > Should modify version 3.0 and 2.0 as well. done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [cherry-pick](branch-3.0) Pick "[Feature](schema change) Support add column bitmap with default value bitmap_empty (#42331)" [doris]
doris-robot commented on PR #42701: URL: https://github.com/apache/doris/pull/42701#issuecomment-2452360055 TPC-H: Total hot run time: 40311 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 60707f8fe321887336bab6f6451e1ef753dd7dd7, data reload: false -- Round 1 -- q1 17572 739072397239 q2 2054154 146 146 q3 10722 144611271127 q4 10565 756 691 691 q5 7730279827652765 q6 230 149 144 144 q7 973 620 599 599 q8 9554189619891896 q9 8034635963576357 q10 6957226922802269 q11 464 260 256 256 q12 400 214 208 208 q13 17785 294229122912 q14 250 214 219 214 q15 567 506 507 506 q16 682 602 601 601 q17 964 572 555 555 q18 7191660865736573 q19 1370105410111011 q20 459 191 201 191 q21 3827326630713071 q22 10831014980 980 Total cold run time: 109433 ms Total hot run time: 40311 ms - Round 2, with runtime_filter_mode=off - q1 7258715072647150 q2 330 234 227 227 q3 2852285428512851 q4 1994180017911791 q5 5622562256935622 q6 225 141 147 141 q7 2185180117601760 q8 3269351534403440 q9 8714878087748774 q10 3499346434923464 q11 597 489 506 489 q12 808 588 599 588 q13 16437 314730573057 q14 313 280 266 266 q15 551 518 516 516 q16 707 667 654 654 q17 1844161315701570 q18 8170781474507450 q19 4079163315191519 q20 2085182618321826 q21 5356530452585258 q22 1125102310011001 Total cold run time: 78020 ms Total hot run time: 59414 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](index compaction)Skip writing terms with a doc frequency of 0 [doris]
doris-robot commented on PR #43113: URL: https://github.com/apache/doris/pull/43113#issuecomment-2452069329 TPC-H: Total hot run time: 41525 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 9456193a1494a2ab26d5ec7d468eae51c69bb217, data reload: false -- Round 1 -- q1 17610 768473037303 q2 2053179 163 163 q3 10557 108011491080 q4 10250 862 835 835 q5 7770307331093073 q6 235 150 150 150 q7 1021600 598 598 q8 9346192620291926 q9 6605642364956423 q10 7081242224172417 q11 458 259 263 259 q12 411 220 205 205 q13 17766 299530142995 q14 244 222 209 209 q15 579 531 516 516 q16 627 591 587 587 q17 1001572 553 553 q18 7419677067586758 q19 1324100710881007 q20 496 182 181 181 q21 4104342732513251 q22 1149103610411036 Total cold run time: 108106 ms Total hot run time: 41525 ms - Round 2, with runtime_filter_mode=off - q1 7343730073077300 q2 324 236 224 224 q3 3068295729982957 q4 2111184817951795 q5 5752580458395804 q6 229 144 141 141 q7 2281183618041804 q8 3411345135443451 q9 8979894589248924 q10 3613357535663566 q11 608 521 509 509 q12 821 622 636 622 q13 11795 320332653203 q14 298 280 280 280 q15 573 518 518 518 q16 698 627 652 627 q17 1871162516381625 q18 8411781877367736 q19 1730162315691569 q20 2090186518841865 q21 5600557154485448 q22 1220109910771077 Total cold run time: 72826 ms Total hot run time: 61045 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](index compaction)Skip writing terms with a doc frequency of 0 [doris]
doris-robot commented on PR #43113: URL: https://github.com/apache/doris/pull/43113#issuecomment-2452090891 TPC-DS: Total hot run time: 197939 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 9456193a1494a2ab26d5ec7d468eae51c69bb217, data reload: false query1 1211897 876 876 query2 6237221521762176 query3 10911 401841764018 query4 68220 29363 23752 23752 query5 4973469 459 459 query6 419 193 173 173 query7 6363300 296 296 query8 315 238 235 235 query9 9331275327592753 query10 471 282 258 258 query11 17811 15853 16020 15853 query12 161 117 104 104 query13 1563450 443 443 query14 11383 728974057289 query15 199 179 175 175 query16 7286488 499 488 query17 1068588 582 582 query18 1803322 309 309 query19 205 168 158 158 query20 123 116 115 115 query21 218 107 107 107 query22 5119480147904790 query23 35607 34038 34109 34038 query24 5947273427952734 query25 520 407 397 397 query26 658 161 158 158 query27 1700287 290 287 query28 4023244224172417 query29 677 434 425 425 query30 235 155 158 155 query31 986 782 783 782 query32 67 59 58 58 query33 451 276 282 276 query34 907 514 506 506 query35 851 754 730 730 query36 1070972 930 930 query37 116 75 77 75 query38 4355424942264226 query39 1507145514421442 query40 203 101 104 101 query41 50 47 46 46 query42 114 98 98 98 query43 549 508 515 508 query44 1187820 822 820 query45 190 166 176 166 query46 1121689 690 689 query47 1931185518531853 query48 424 331 334 331 query49 743 393 416 393 query50 807 403 411 403 query51 7369714071867140 query52 101 91 88 88 query53 257 184 187 184 query54 532 421 418 418 query55 74 79 74 74 query56 254 239 242 239 query57 1297118411621162 query58 216 207 206 206 query59 3215312929742974 query60 272 254 246 246 query61 108 101 107 101 query62 795 696 670 670 query63 218 193 195 193 query64 1366633 597 597 query65 3311321331763176 query66 705 354 309 309 query67 16006 15760 15863 15760 query68 4491551 582 551 query69 419 252 259 252 query70 1215114911491149 query71 366 243 251 243 query72 6223406339603960 query73 756 362 363 362 query74 10387 931489958995 query75 3388267527132675 query76 1793103410381034 query77 474 278 274 274 query78 10570 944093329332 query79 1099583 586 583 query80 768 421 452 421 query81 512 243 239 239 query82 985 120 122 120 query83 220 146 144 144 query84 282 73 70 70 query85 850 290 295 290 query86 319 293 255 255 query87 5005467047054670 query88 3490217221702170 query89 408 291 281 281 query90 2065189 189 189 query91 136 97 107 97 query92 59 52 47 47 query93 1071526 531 526 query94 776 281 288 281 query95 333 266 255 255 query96 615 276 294 276 query97 2922269427432694 query98 218 205 191 191 query99 1718133813231323 Total cold run time: 322415 ms Total hot run time: 197939 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] branch-3.0: [test](stream load) add cases to ensure the correctness of the document [doris]
github-actions[bot] commented on PR #43111: URL: https://github.com/apache/doris/pull/43111#issuecomment-2451900671 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
