Re: [I] [Bug] Doris Be node crash ,version:doris-1.2.7 [doris]

2023-10-21 Thread via GitHub


fsilent commented on issue #24228:
URL: https://github.com/apache/doris/issues/24228#issuecomment-1773702069

   hello, have you solved this problem?  @liuchunhua 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [regresstion-test](jdbc catalog)Mariadb compatible test [doris]

2023-10-21 Thread via GitHub


zy-kkk commented on PR #25664:
URL: https://github.com/apache/doris/pull/25664#issuecomment-1773719659

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [build] Update build-extension.yml [doris]

2023-10-21 Thread via GitHub


zy-kkk merged PR #25721:
URL: https://github.com/apache/doris/pull/25721


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch master updated: [build](extension) Update build-extension.yml (#25721)

2023-10-21 Thread zykkk
This is an automated email from the ASF dual-hosted git repository.

zykkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 387a9c74489 [build](extension) Update build-extension.yml (#25721)
387a9c74489 is described below

commit 387a9c74489c47930d3bb2506c70fdd06f0cefde
Author: Jeffrey 
AuthorDate: Sat Oct 21 16:52:55 2023 +0800

[build](extension) Update build-extension.yml (#25721)
---
 .github/workflows/build-extension.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.github/workflows/build-extension.yml 
b/.github/workflows/build-extension.yml
index 4a9fc5d38ef..d58e21a75dc 100644
--- a/.github/workflows/build-extension.yml
+++ b/.github/workflows/build-extension.yml
@@ -100,6 +100,7 @@ jobs:
 run: |
   git clone https://github.com/apache/doris-website.git website
   cd website
+  echo "[\"current\"]" > versions.json 
   mkdir -p docs
   cp -R ../docs/en/docs/* docs/
   cp -R ../docs/sidebars.json sidebars.json


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch branch-2.0 updated: [fix](ut) fix fe ut for branch-2.0 (#25719)

2023-10-21 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 3389f3f9d1d [fix](ut) fix fe ut for branch-2.0 (#25719)
3389f3f9d1d is described below

commit 3389f3f9d1d17f3018b0bc20970298a04a380324
Author: AKIRA <33112463+kikyou1...@users.noreply.github.com>
AuthorDate: Sat Oct 21 18:31:36 2023 +0800

[fix](ut) fix fe ut for branch-2.0 (#25719)
---
 .../doris/statistics/AnalysisManagerTest.java  | 62 ++
 1 file changed, 39 insertions(+), 23 deletions(-)

diff --git 
a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisManagerTest.java 
b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisManagerTest.java
index 636e32ea4e1..c995710da44 100644
--- 
a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisManagerTest.java
+++ 
b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisManagerTest.java
@@ -25,6 +25,8 @@ import org.apache.doris.catalog.Column;
 import org.apache.doris.catalog.OlapTable;
 import org.apache.doris.catalog.PrimitiveType;
 import org.apache.doris.common.DdlException;
+import org.apache.doris.statistics.AnalysisInfo.AnalysisType;
+import org.apache.doris.statistics.AnalysisInfo.JobType;
 import org.apache.doris.statistics.AnalysisInfo.ScheduleType;
 import org.apache.doris.statistics.util.StatisticsUtil;
 
@@ -45,10 +47,11 @@ import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
 
+// CHECKSTYLE OFF
 public class AnalysisManagerTest {
 @Test
 public void testUpdateTaskStatus(@Mocked BaseAnalysisTask task1,
- @Mocked BaseAnalysisTask task2) {
+@Mocked BaseAnalysisTask task2) {
 
 new MockUp() {
 @Mock
@@ -61,12 +64,22 @@ public class AnalysisManagerTest {
 
 };
 
+new MockUp() {
+@Mock
+public String toString() {
+return "";
+}
+};
+
 AnalysisInfo job = new AnalysisInfoBuilder().setJobId(1)
-
.setState(AnalysisState.PENDING).setJobType(AnalysisInfo.JobType.MANUAL).build();
+
.setState(AnalysisState.PENDING).setAnalysisType(AnalysisType.FUNDAMENTALS)
+.setJobType(AnalysisInfo.JobType.MANUAL).build();
 AnalysisInfo taskInfo1 = new AnalysisInfoBuilder().setJobId(1)
-.setTaskId(2).setState(AnalysisState.PENDING).build();
+
.setTaskId(2).setJobType(JobType.MANUAL).setAnalysisType(AnalysisType.FUNDAMENTALS)
+.setState(AnalysisState.PENDING).build();
 AnalysisInfo taskInfo2 = new AnalysisInfoBuilder().setJobId(1)
-.setTaskId(3).setState(AnalysisState.PENDING).build();
+
.setTaskId(3).setAnalysisType(AnalysisType.FUNDAMENTALS).setJobType(JobType.MANUAL)
+.setState(AnalysisState.PENDING).build();
 AnalysisManager manager = new AnalysisManager();
 manager.replayCreateAnalysisJob(job);
 manager.replayCreateAnalysisTask(taskInfo1);
@@ -139,15 +152,15 @@ public class AnalysisManagerTest {
 add("p2");
 }
 }), new ArrayList() {
-{
-add("c1");
-add("c2");
-}
-}, new AnalyzeProperties(new HashMap() {
-{
-put(AnalyzeProperties.PROPERTY_SYNC, "true");
-}
-}));
+{
+add("c1");
+add("c2");
+}
+}, new AnalyzeProperties(new HashMap() {
+{
+put(AnalyzeProperties.PROPERTY_SYNC, "true");
+}
+}));
 
 AnalysisManager analysisManager = new AnalysisManager();
 
Assertions.assertNull(analysisManager.buildAndAssignJob(analyzeTblStmt));
@@ -228,16 +241,16 @@ public class AnalysisManagerTest {
 add("p2");
 }
 }), new ArrayList() {
-{
-add("c1");
-add("c2");
-}
-}, new AnalyzeProperties(new HashMap() {
-{
-put(AnalyzeProperties.PROPERTY_SYNC, "false");
-put(AnalyzeProperties.PROPERTY_PERIOD_SECONDS, "100");
-}
-}));
+{
+add("c1");
+add("c2");
+}
+}, new AnalyzeProperties(new HashMap() {
+{
+put(AnalyzeProperties.PROPERTY_SYNC, "false");
+put(AnalyzeProperties.PROPERTY_PERIOD_SECONDS, "100");
+}
+}));
 AnalysisManager analysisManager = new AnalysisManager(

Re: [PR] [fix](stats) Fix ut [doris]

2023-10-21 Thread via GitHub


xiaokang merged PR #25719:
URL: https://github.com/apache/doris/pull/25719


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](CANCEL-ALTER-SYSTEM)decommission backend by ids [doris]

2023-10-21 Thread via GitHub


liugddx commented on PR #25441:
URL: https://github.com/apache/doris/pull/25441#issuecomment-1773748000

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](CANCEL-ALTER-SYSTEM)decommission backend by ids [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #25441:
URL: https://github.com/apache/doris/pull/25441#issuecomment-1773761297

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.39 seconds
stream load tsv:  551 seconds loaded 74807831229 Bytes, about 129 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  29.0 seconds inserted 1000 Rows, about 
344K ops/s
storage size: 17162008636 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [I] [Bug] Doris Be node crash ,version:doris-1.2.7 [doris]

2023-10-21 Thread via GitHub


liuchunhua commented on issue #24228:
URL: https://github.com/apache/doris/issues/24228#issuecomment-1773761952

   not yet


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](paimon)paimon catalog supports complex types [doris]

2023-10-21 Thread via GitHub


DongLiang-0 commented on PR #25364:
URL: https://github.com/apache/doris/pull/25364#issuecomment-1773771412

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](paimon)paimon catalog supports complex types [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #25364:
URL: https://github.com/apache/doris/pull/25364#issuecomment-1773781759

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.81 seconds
stream load tsv:  555 seconds loaded 74807831229 Bytes, about 128 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  33 seconds loaded 861443392 Bytes, about 24 
MB/s
insert into select:  28.8 seconds inserted 1000 Rows, about 
347K ops/s
storage size: 17162015400 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [refactor](streamload) refactor stream load executor [doris]

2023-10-21 Thread via GitHub


HHoflittlefish777 commented on PR #25615:
URL: https://github.com/apache/doris/pull/25615#issuecomment-1773791004

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [refactor](streamload) refactor stream load executor [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25615:
URL: https://github.com/apache/doris/pull/25615#issuecomment-1773793191

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [refactor](streamload) refactor stream load executor [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25615:
URL: https://github.com/apache/doris/pull/25615#issuecomment-1773793855

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [refactor](streamload) refactor stream load executor [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #25615:
URL: https://github.com/apache/doris/pull/25615#issuecomment-1773798091

   TeamCity be ut coverage result:
Function Coverage: 37.05% (8303/22410) 
Line Coverage: 29.21% (66693/228326)
Region Coverage: 27.84% (34626/124360)
Branch Coverage: 24.43% (17586/71996)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/aa21dc81218a83cf2addf9a11c1c0162676a1555_aa21dc81218a83cf2addf9a11c1c0162676a1555/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [refactor](streamload) refactor stream load executor [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #25615:
URL: https://github.com/apache/doris/pull/25615#issuecomment-1773803643

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.67 seconds
stream load tsv:  568 seconds loaded 74807831229 Bytes, about 125 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  28.9 seconds inserted 1000 Rows, about 
346K ops/s
storage size: 17162439292 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [enhancement](recover) support skipping missing version in select by session variable [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25654:
URL: https://github.com/apache/doris/pull/25654#issuecomment-1773809247

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [doc](pick)support microseconds/milliseconds func [doris]

2023-10-21 Thread via GitHub


xiaokang merged PR #25245:
URL: https://github.com/apache/doris/pull/25245


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch branch-2.0 updated: [doc](pick)support microseconds/milliseconds func (#25245)

2023-10-21 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new a5d5a8bbe03 [doc](pick)support microseconds/milliseconds func (#25245)
a5d5a8bbe03 is described below

commit a5d5a8bbe0366b5f3eb1943929edf0233769062d
Author: Liqf <109049295+lemonlit...@users.noreply.github.com>
AuthorDate: Sat Oct 21 22:33:16 2023 +0800

[doc](pick)support microseconds/milliseconds func (#25245)
---
 docs/sidebars.json | 5 +
 1 file changed, 5 insertions(+)

diff --git a/docs/sidebars.json b/docs/sidebars.json
index fb8982d199f..089f0022d75 100644
--- a/docs/sidebars.json
+++ b/docs/sidebars.json
@@ -360,6 +360,11 @@
 
"sql-manual/sql-functions/date-time-functions/date_format",
 
"sql-manual/sql-functions/date-time-functions/datediff",
 
"sql-manual/sql-functions/date-time-functions/microseconds_add",
+
"sql-manual/sql-functions/date-time-functions/microseconds-diff",
+
"sql-manual/sql-functions/date-time-functions/microseconds-sub",
+
"sql-manual/sql-functions/date-time-functions/milliseconds-add",
+
"sql-manual/sql-functions/date-time-functions/milliseconds-diff",
+
"sql-manual/sql-functions/date-time-functions/milliseconds-sub",
 
"sql-manual/sql-functions/date-time-functions/minutes_add",
 
"sql-manual/sql-functions/date-time-functions/minutes_diff",
 
"sql-manual/sql-functions/date-time-functions/minutes_sub",


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](ColdHeatSeperation) Support to upload cold data to HDFS [doris]

2023-10-21 Thread via GitHub


morningman commented on PR #22048:
URL: https://github.com/apache/doris/pull/22048#issuecomment-177386

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [chore](log) Print correct queryId & instance & [doris]

2023-10-21 Thread via GitHub


xiaokang commented on PR #25532:
URL: https://github.com/apache/doris/pull/25532#issuecomment-1773811397

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feat](stats) Enable auto sample by default [doris]

2023-10-21 Thread via GitHub


xiaokang commented on PR #25607:
URL: https://github.com/apache/doris/pull/25607#issuecomment-1773811711

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](nereids) runtime filter generator meet npe when process set operation [doris]

2023-10-21 Thread via GitHub


xiaokang commented on PR #25612:
URL: https://github.com/apache/doris/pull/25612#issuecomment-1773811843

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [pick](Branch2.0) fix row count unconsistent when join ordering (#24589) [doris]

2023-10-21 Thread via GitHub


xiaokang commented on PR #25666:
URL: https://github.com/apache/doris/pull/25666#issuecomment-1773812135

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [pick](Branch2.0) generate left deep tree when stats is unknown [doris]

2023-10-21 Thread via GitHub


xiaokang commented on PR #25702:
URL: https://github.com/apache/doris/pull/25702#issuecomment-1773812298

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](nereids)pick part of "[opt](nereids) enable runtime filter prune by default #24717" [doris]

2023-10-21 Thread via GitHub


xiaokang commented on PR #25642:
URL: https://github.com/apache/doris/pull/25642#issuecomment-1773811996

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [opt](profile) set datalake profile level as 1 [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25686:
URL: https://github.com/apache/doris/pull/25686#issuecomment-1773812457

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [improvement](sync version) fe sync version with be [doris]

2023-10-21 Thread via GitHub


xiaokang commented on PR #25718:
URL: https://github.com/apache/doris/pull/25718#issuecomment-1773812469

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [opt](profile) set datalake profile level as 1 [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25686:
URL: https://github.com/apache/doris/pull/25686#issuecomment-1773812477

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](ColdHeatSeperation) Support to upload cold data to HDFS [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #22048:
URL: https://github.com/apache/doris/pull/22048#issuecomment-1773813573

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [chore](log) Print correct queryId & instance & [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25532:
URL: https://github.com/apache/doris/pull/25532#issuecomment-1773814005

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [improvement](sync version) fe sync version with be [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25718:
URL: https://github.com/apache/doris/pull/25718#issuecomment-1773814785

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](Outfile) Export `DateTimev2` type of doris to ORC's `TimeStamp` type [doris]

2023-10-21 Thread via GitHub


morningman commented on code in PR #25470:
URL: https://github.com/apache/doris/pull/25470#discussion_r1367740540


##
be/src/vec/runtime/vorc_transformer.cpp:
##
@@ -94,7 +94,10 @@ VOrcTransformer::VOrcTransformer(doris::io::FileWriter* 
file_writer,
 : VFileFormatTransformer(output_vexpr_ctxs, output_object_data),
   _file_writer(file_writer),
   _write_options(new orc::WriterOptions()),
-  _schema_str(schema) {}
+  _schema_str(schema) {
+_write_options->setTimezoneName("Asia/Shanghai");

Review Comment:
   Should it be a fixed value?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [opt](filecache) do not sync segment data into storage system [doris]

2023-10-21 Thread via GitHub


morningman commented on PR #25691:
URL: https://github.com/apache/doris/pull/25691#issuecomment-1773817528

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [opt](filecache) do not sync segment data into storage system [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25691:
URL: https://github.com/apache/doris/pull/25691#issuecomment-1773818189

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [opt](filecache) do not sync segment data into storage system [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25691:
URL: https://github.com/apache/doris/pull/25691#issuecomment-1773818201

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](ColdHeatSeperation) Support to upload cold data to HDFS [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #22048:
URL: https://github.com/apache/doris/pull/22048#issuecomment-1773821377

   TeamCity be ut coverage result:
Function Coverage: 37.05% (8303/22410) 
Line Coverage: 29.20% (66686/228341)
Region Coverage: 27.84% (34624/124371)
Branch Coverage: 24.42% (17586/72002)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/91d8ff73cc2d59f29861a1c84e14936093a88471_91d8ff73cc2d59f29861a1c84e14936093a88471/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [improvement](auth) support show all property [doris]

2023-10-21 Thread via GitHub


morningman commented on code in PR #25645:
URL: https://github.com/apache/doris/pull/25645#discussion_r1367742106


##
docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-PROPERTY.md:
##
@@ -62,6 +66,26 @@ mysql> show property like'%connection%';
 
 Attribute value.
 
+
+```sql
+mysql> show all property like "%connection%";

Review Comment:
   better use `show all properties`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](auth)Fix no priv to insert to mysql catalog [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25678:
URL: https://github.com/apache/doris/pull/25678#issuecomment-1773823926

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](auth)Fix no priv to insert to mysql catalog [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25678:
URL: https://github.com/apache/doris/pull/25678#issuecomment-1773823939

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [improvement](scanner_schedule) reduce memory consumption of scanner #24199 [doris]

2023-10-21 Thread via GitHub


xiaokang commented on PR #25547:
URL: https://github.com/apache/doris/pull/25547#issuecomment-1773825452

   convert it to draft since an bug is reported related to this pr on master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](dynamic) Fix error reporting when dynamic partition properties contain incorrect attributes [doris]

2023-10-21 Thread via GitHub


Zhiyu-h commented on PR #25373:
URL: https://github.com/apache/doris/pull/25373#issuecomment-1773825590

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](nereids)(create-table) fix bug that replication num is not set when create table with no property [doris]

2023-10-21 Thread via GitHub


morningman merged PR #25651:
URL: https://github.com/apache/doris/pull/25651


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch master updated: [fix](nereids)(create-table) fix bug that replication num is not set when create table with no property (#25651)

2023-10-21 Thread morningman
This is an automated email from the ASF dual-hosted git repository.

morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
 new 13780e4827d [fix](nereids)(create-table) fix bug that replication num 
is not set when create table with no property (#25651)
13780e4827d is described below

commit 13780e4827db38494f2f3fb168f3195163a18fc1
Author: Mingyu Chen 
AuthorDate: Sat Oct 21 23:15:08 2023 +0800

[fix](nereids)(create-table) fix bug that replication num is not set when 
create table with no property (#25651)

When executing create partitioned table with Nereids, and replication_num 
property is not set,
the replication number will be 0, so the tablet will has no replica.
---
 .../org/apache/doris/analysis/CreateTableStmt.java |  75 +-
 .../apache/doris/common/util/PropertyAnalyzer.java |  72 ++
 .../doris/nereids/parser/LogicalPlanBuilder.java   |  11 ++-
 .../trees/plans/commands/info/CreateTableInfo.java |  45 ++---
 .../plans/commands/info/FixedRangePartition.java   |  10 +-
 .../trees/plans/commands/info/InPartition.java |   9 +-
 .../plans/commands/info/LessThanPartition.java |  10 +-
 .../plans/commands/info/PartitionDefinition.java   |  22 -
 .../trees/plans/commands/info/StepPartition.java   |   9 +-
 .../org/apache/doris/catalog/CreateTableTest.java  | 110 +
 .../org/apache/doris/common/ExceptionChecker.java  |   1 +
 .../apache/doris/utframe/TestWithFeService.java|  37 +--
 12 files changed, 240 insertions(+), 171 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java 
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
index a03884f5c6e..f56de294d98 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java
@@ -20,13 +20,11 @@ package org.apache.doris.analysis;
 import org.apache.doris.analysis.IndexDef.IndexType;
 import org.apache.doris.catalog.AggregateType;
 import org.apache.doris.catalog.Column;
-import org.apache.doris.catalog.DatabaseIf;
 import org.apache.doris.catalog.DistributionInfo;
 import org.apache.doris.catalog.Env;
 import org.apache.doris.catalog.Index;
 import org.apache.doris.catalog.KeysType;
 import org.apache.doris.catalog.PrimitiveType;
-import org.apache.doris.catalog.ReplicaAllocation;
 import org.apache.doris.catalog.Type;
 import org.apache.doris.common.AnalysisException;
 import org.apache.doris.common.Config;
@@ -41,7 +39,6 @@ import org.apache.doris.common.util.ParseUtil;
 import org.apache.doris.common.util.PrintableMap;
 import org.apache.doris.common.util.PropertyAnalyzer;
 import org.apache.doris.common.util.Util;
-import org.apache.doris.datasource.CatalogIf;
 import org.apache.doris.external.elasticsearch.EsUtil;
 import org.apache.doris.mysql.privilege.PrivPredicate;
 import org.apache.doris.qe.ConnectContext;
@@ -51,7 +48,6 @@ import com.google.common.collect.Lists;
 import com.google.common.collect.Maps;
 import com.google.common.collect.Sets;
 import org.apache.commons.collections.CollectionUtils;
-import org.apache.commons.lang3.StringUtils;
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
 
@@ -559,7 +555,8 @@ public class CreateTableStmt extends DdlStmt {
 
 if (engineName.equals("olap")) {
 // before analyzing partition, handle the replication allocation 
info
-properties = rewriteReplicaAllocationProperties(properties);
+properties = PropertyAnalyzer.rewriteReplicaAllocationProperties(
+tableName.getCtl(), tableName.getDb(), properties);
 // analyze partition
 if (partitionDesc != null) {
 if (partitionDesc instanceof ListPartitionDesc || 
partitionDesc instanceof RangePartitionDesc
@@ -650,74 +647,6 @@ public class CreateTableStmt extends DdlStmt {
 }
 }
 
-private Map rewriteReplicaAllocationProperties(Map properties)
-throws AnalysisException {
-if (Config.force_olap_table_replication_num <= 0) {
-return rewriteReplicaAllocationPropertiesByDatabase(properties);
-}
-// if force_olap_table_replication_num is set, use this value to 
rewrite the replication_num or
-// replication_allocation properties
-Map newProperties = properties;
-if (newProperties == null) {
-newProperties = Maps.newHashMap();
-}
-boolean rewrite = false;
-if 
(newProperties.containsKey(PropertyAnalyzer.PROPERTIES_REPLICATION_NUM)) {
-newProperties.put(PropertyAnalyzer.PROPERTIES_REPLICATION_NUM,
-String.valueOf(Config.force_olap_table_replication_num));
-rewrite = true;
-}
-if 
(newProp

Re: [PR] [feat](stats) Enable auto sample by default [doris]

2023-10-21 Thread via GitHub


hello-stephen commented on PR #25607:
URL: https://github.com/apache/doris/pull/25607#issuecomment-1773827449

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.26 seconds
stream load tsv:  573 seconds loaded 74807831229 Bytes, about 124 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  31 seconds loaded 861443392 Bytes, about 26 
MB/s
insert into select:  29.5 seconds inserted 1000 Rows, about 
338K ops/s
storage size: 17163856179 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [pick](Branch2.0) fix row count unconsistent when join ordering (#24589) [doris]

2023-10-21 Thread via GitHub


hello-stephen commented on PR #25666:
URL: https://github.com/apache/doris/pull/25666#issuecomment-1773827506

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.06 seconds
stream load tsv:  576 seconds loaded 74807831229 Bytes, about 123 
MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  33 seconds loaded 861443392 Bytes, about 24 
MB/s
insert into select:  29.9 seconds inserted 1000 Rows, about 
334K ops/s
storage size: 17167870587 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](nereids) runtime filter generator meet npe when process set operation [doris]

2023-10-21 Thread via GitHub


hello-stephen commented on PR #25612:
URL: https://github.com/apache/doris/pull/25612#issuecomment-1773827571

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.96 seconds
stream load tsv:  577 seconds loaded 74807831229 Bytes, about 123 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  31 seconds loaded 861443392 Bytes, about 26 
MB/s
insert into select:  29.5 seconds inserted 1000 Rows, about 
338K ops/s
storage size: 17168027630 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [chore](log) Print correct queryId & instance & [doris]

2023-10-21 Thread via GitHub


hello-stephen commented on PR #25532:
URL: https://github.com/apache/doris/pull/25532#issuecomment-1773827747

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.74 seconds
stream load tsv:  574 seconds loaded 74807831229 Bytes, about 124 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  29.5 seconds inserted 1000 Rows, about 
338K ops/s
storage size: 17163160328 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](ColdHeatSeperation) Support to upload cold data to HDFS [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #22048:
URL: https://github.com/apache/doris/pull/22048#issuecomment-1773827823

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.85 seconds
stream load tsv:  553 seconds loaded 74807831229 Bytes, about 129 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  28.7 seconds inserted 1000 Rows, about 
348K ops/s
storage size: 17162151206 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [pick](Branch2.0) generate left deep tree when stats is unknown [doris]

2023-10-21 Thread via GitHub


hello-stephen commented on PR #25702:
URL: https://github.com/apache/doris/pull/25702#issuecomment-1773831287

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.95 seconds
stream load tsv:  576 seconds loaded 74807831229 Bytes, about 123 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  31 seconds loaded 861443392 Bytes, about 26 
MB/s
insert into select:  29.2 seconds inserted 1000 Rows, about 
342K ops/s
storage size: 17162441392 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [opt](filecache) do not sync segment data into storage system [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #25691:
URL: https://github.com/apache/doris/pull/25691#issuecomment-1773831836

   TeamCity be ut coverage result:
Function Coverage: 37.04% (8301/22410) 
Line Coverage: 29.20% (66676/228328)
Region Coverage: 27.84% (34622/124363)
Branch Coverage: 24.43% (17587/71998)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/62048a06842b474bf86fe51c6e1a974bbc0c658b_62048a06842b474bf86fe51c6e1a974bbc0c658b/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix](exec) Fix date dict infinite loop. [doris]

2023-10-21 Thread via GitHub


kaka11chen commented on PR #25570:
URL: https://github.com/apache/doris/pull/25570#issuecomment-1773837048

   run buildalll


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](alter backends)backend modify & drop & decommission by ids [doris]

2023-10-21 Thread via GitHub


liugddx commented on PR #25444:
URL: https://github.com/apache/doris/pull/25444#issuecomment-1773838359

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix](exec) Fix date dict infinite loop. [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on code in PR #25570:
URL: https://github.com/apache/doris/pull/25570#discussion_r1367751431


##
be/test/vec/runtime/vdatetime_value_test.cpp:
##
@@ -570,4 +570,172 @@ TEST(VDateTimeValueTest, date_v2_to_string_test) {
 }
 }
 
+TEST(VDateTimeValueTest, date_v2_daynr_test) {
+{
+DateV2Value date_v2;
+// 1970/01/01
+EXPECT_TRUE(date_v2.get_date_from_daynr(719528));
+EXPECT_TRUE(date_v2.year() == 1970);
+EXPECT_TRUE(date_v2.month() == 1);
+EXPECT_TRUE(date_v2.day() == 1);
+EXPECT_TRUE(date_v2.hour() == 0);
+EXPECT_TRUE(date_v2.minute() == 0);
+EXPECT_TRUE(date_v2.second() == 0);
+EXPECT_TRUE(date_v2.microsecond() == 0);
+EXPECT_TRUE(doris::calc_daynr(1970, 1, 1) == 719528);
+EXPECT_TRUE(date_day_offset_dict::get().get_dict_init());
+EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_calc_daynr(1970));
+
EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_daynr_to_date(719528));
+}
+
+{
+DateV2Value date_v2;
+// 1969/12/31
+EXPECT_TRUE(date_v2.get_date_from_daynr(719527));
+EXPECT_TRUE(date_v2.year() == 1969);
+EXPECT_TRUE(date_v2.month() == 12);
+EXPECT_TRUE(date_v2.day() == 31);
+EXPECT_TRUE(date_v2.hour() == 0);
+EXPECT_TRUE(date_v2.minute() == 0);
+EXPECT_TRUE(date_v2.second() == 0);
+EXPECT_TRUE(date_v2.microsecond() == 0);
+EXPECT_TRUE(doris::calc_daynr(1969, 12, 31) == 719527);
+EXPECT_TRUE(date_day_offset_dict::get().get_dict_init());
+EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_calc_daynr(1969));
+
EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_daynr_to_date(719527));
+}
+
+{
+DateV2Value date_v2;
+// 1900/01/01
+EXPECT_TRUE(date_v2.get_date_from_daynr(693961));
+EXPECT_TRUE(date_v2.year() == 1900);
+EXPECT_TRUE(date_v2.month() == 1);
+EXPECT_TRUE(date_v2.day() == 1);
+EXPECT_TRUE(date_v2.hour() == 0);
+EXPECT_TRUE(date_v2.minute() == 0);
+EXPECT_TRUE(date_v2.second() == 0);
+EXPECT_TRUE(date_v2.microsecond() == 0);
+EXPECT_TRUE(doris::calc_daynr(1900, 1, 1) == 693961);
+EXPECT_TRUE(date_day_offset_dict::get().get_dict_init());
+EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_calc_daynr(1900));
+
EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_daynr_to_date(693961));
+}
+
+{
+DateV2Value date_v2;
+// 1899/12/31
+EXPECT_TRUE(date_v2.get_date_from_daynr(693960));
+EXPECT_TRUE(date_v2.year() == 1899);
+EXPECT_TRUE(date_v2.month() == 12);
+EXPECT_TRUE(date_v2.day() == 31);
+EXPECT_TRUE(date_v2.hour() == 0);
+EXPECT_TRUE(date_v2.minute() == 0);
+EXPECT_TRUE(date_v2.second() == 0);
+EXPECT_TRUE(date_v2.microsecond() == 0);
+EXPECT_TRUE(doris::calc_daynr(1899, 12, 31) == 693960);
+EXPECT_TRUE(date_day_offset_dict::get().get_dict_init());
+
EXPECT_FALSE(date_day_offset_dict::get().can_speed_up_calc_daynr(1899));
+
EXPECT_FALSE(date_day_offset_dict::get().can_speed_up_daynr_to_date(693960));
+}
+
+{
+DateV2Value date_v2;
+// 2039/12/31
+EXPECT_TRUE(date_v2.get_date_from_daynr(745094));
+EXPECT_TRUE(date_v2.year() == 2039);
+EXPECT_TRUE(date_v2.month() == 12);
+EXPECT_TRUE(date_v2.day() == 31);
+EXPECT_TRUE(date_v2.hour() == 0);
+EXPECT_TRUE(date_v2.minute() == 0);
+EXPECT_TRUE(date_v2.second() == 0);
+EXPECT_TRUE(date_v2.microsecond() == 0);
+EXPECT_TRUE(doris::calc_daynr(2039, 12, 31) == 745094);
+EXPECT_TRUE(date_day_offset_dict::get().get_dict_init());
+EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_calc_daynr(2039));
+
EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_daynr_to_date(745094));
+}
+
+{
+DateV2Value date_v2;
+// 2040/01/01
+EXPECT_TRUE(date_v2.get_date_from_daynr(745095));
+EXPECT_TRUE(date_v2.year() == 2040);
+EXPECT_TRUE(date_v2.month() == 1);
+EXPECT_TRUE(date_v2.day() == 1);
+EXPECT_TRUE(date_v2.hour() == 0);
+EXPECT_TRUE(date_v2.minute() == 0);
+EXPECT_TRUE(date_v2.second() == 0);
+EXPECT_TRUE(date_v2.microsecond() == 0);
+EXPECT_TRUE(doris::calc_daynr(2040, 01, 01) == 745095);
+EXPECT_TRUE(date_day_offset_dict::get().get_dict_init());
+
EXPECT_FALSE(date_day_offset_dict::get().can_speed_up_calc_daynr(2040));
+
EXPECT_FALSE(date_day_offset_dict::get().can_speed_up_daynr_to_date(745095));
+}
+
+{
+DateV2Value date_v2;
+// /01/01
+EXPECT_TRUE(date_v2.get_date_from_daynr(1));
+EXPECT_TRUE(date_v2.year() == 0);
+EXPECT_TRUE(date_v2.month() == 1);
+   

Re: [PR] [chore](log) Print correct queryId & instance & [doris]

2023-10-21 Thread via GitHub


xiaokang merged PR #25532:
URL: https://github.com/apache/doris/pull/25532


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch branch-2.0 updated: [chore](log) Print correct queryId & instance (#25532)

2023-10-21 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 6f9e39e4093 [chore](log) Print correct queryId & instance  (#25532)
6f9e39e4093 is described below

commit 6f9e39e40930a90d67e2574fa89832f905dbc210
Author: zhiqiang 
AuthorDate: Sat Oct 21 11:42:59 2023 -0500

[chore](log) Print correct queryId & instance  (#25532)
---
 be/src/pipeline/pipeline_fragment_context.cpp |  4 ++--
 be/src/runtime/fragment_mgr.cpp   |  4 ++--
 be/src/runtime/plan_fragment_executor.cpp | 10 +-
 3 files changed, 9 insertions(+), 9 deletions(-)

diff --git a/be/src/pipeline/pipeline_fragment_context.cpp 
b/be/src/pipeline/pipeline_fragment_context.cpp
index f18f437ba47..cd529ee835d 100644
--- a/be/src/pipeline/pipeline_fragment_context.cpp
+++ b/be/src/pipeline/pipeline_fragment_context.cpp
@@ -211,8 +211,8 @@ Status PipelineFragmentContext::prepare(const 
doris::TPipelineFragmentParams& re
 }
 
 LOG_INFO("PipelineFragmentContext::prepare")
-.tag("query_id", _query_id)
-.tag("instance_id", local_params.fragment_instance_id)
+.tag("query_id", print_id(_query_id))
+.tag("instance_id", print_id(local_params.fragment_instance_id))
 .tag("backend_num", local_params.backend_num)
 .tag("pthread_id", (uintptr_t)pthread_self());
 
diff --git a/be/src/runtime/fragment_mgr.cpp b/be/src/runtime/fragment_mgr.cpp
index 409a32a128d..9f38e91ff15 100644
--- a/be/src/runtime/fragment_mgr.cpp
+++ b/be/src/runtime/fragment_mgr.cpp
@@ -526,8 +526,8 @@ void 
FragmentMgr::_exec_actual(std::shared_ptr exec_state,
 #endif
 
 LOG_INFO(func_name)
-.tag("query_id", exec_state->query_id())
-.tag("instance_id", exec_state->fragment_instance_id())
+.tag("query_id", print_id(exec_state->query_id()))
+.tag("instance_id", print_id(exec_state->fragment_instance_id()))
 .tag("pthread_id", (uintptr_t)pthread_self());
 
 Status st = exec_state->execute();
diff --git a/be/src/runtime/plan_fragment_executor.cpp 
b/be/src/runtime/plan_fragment_executor.cpp
index 77d15938303..de6f5f55810 100644
--- a/be/src/runtime/plan_fragment_executor.cpp
+++ b/be/src/runtime/plan_fragment_executor.cpp
@@ -113,8 +113,8 @@ Status PlanFragmentExecutor::prepare(const 
TExecPlanFragmentParams& request,
 _query_id = params.query_id;
 
 LOG_INFO("PlanFragmentExecutor::prepare")
-.tag("query_id", _query_id)
-.tag("instance_id", params.fragment_instance_id)
+.tag("query_id", print_id(_query_id))
+.tag("instance_id", print_id(params.fragment_instance_id))
 .tag("backend_num", request.backend_num)
 .tag("pthread_id", (uintptr_t)pthread_self());
 // VLOG_CRITICAL << "request:\n" << 
apache::thrift::ThriftDebugString(request);
@@ -249,8 +249,8 @@ Status PlanFragmentExecutor::prepare(const 
TExecPlanFragmentParams& request,
 Status PlanFragmentExecutor::open() {
 int64_t mem_limit = _runtime_state->query_mem_tracker()->limit();
 LOG_INFO("PlanFragmentExecutor::open")
-.tag("query_id", _query_id)
-.tag("instance_id", _runtime_state->fragment_instance_id())
+.tag("query_id", print_id(_query_id))
+.tag("instance_id", 
print_id(_runtime_state->fragment_instance_id()))
 .tag("mem_limit", PrettyPrinter::print(mem_limit, TUnit::BYTES));
 
 // we need to start the profile-reporting thread before calling Open(), 
since it
@@ -488,7 +488,7 @@ void PlanFragmentExecutor::stop_report_thread() {
 
 void PlanFragmentExecutor::cancel(const PPlanFragmentCancelReason& reason, 
const std::string& msg) {
 LOG_INFO("PlanFragmentExecutor::cancel")
-.tag("query_id", _query_id)
+.tag("query_id", print_id(_query_id))
 .tag("instance_id", _runtime_state->fragment_instance_id())
 .tag("reason", reason)
 .tag("error message", msg);


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](nereids) runtime filter generator meet npe when process set operation [doris]

2023-10-21 Thread via GitHub


xiaokang merged PR #25612:
URL: https://github.com/apache/doris/pull/25612


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch branch-2.0 updated: [fix](nereids) runtime filter generator meet npe when process set operation (#25612)

2023-10-21 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 5a21a03b279 [fix](nereids) runtime filter generator meet npe when 
process set operation (#25612)
5a21a03b279 is described below

commit 5a21a03b279a5e7bd85ef2c5a03028fbec3824d4
Author: minghong 
AuthorDate: Sun Oct 22 00:44:18 2023 +0800

[fix](nereids) runtime filter generator meet npe when process set operation 
(#25612)
---
 .../apache/doris/nereids/processor/post/RuntimeFilterGenerator.java | 6 --
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java
 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java
index 6fd2f8838c2..0243326b106 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java
@@ -368,8 +368,10 @@ public class RuntimeFilterGenerator extends 
PlanPostProcessor {
 break;
 }
 }
-Preconditions.checkState(projIndex >= 0
-&& projIndex < project.getProjects().size());
+if (projIndex < 0
+|| projIndex >= project.getProjects().size()) {
+continue;
+}
 
 NamedExpression targetExpr = (NamedExpression) 
project.getProjects().get(projIndex);
 


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](nereids)pick part of "[opt](nereids) enable runtime filter prune by default #24717" [doris]

2023-10-21 Thread via GitHub


xiaokang commented on PR #25642:
URL: https://github.com/apache/doris/pull/25642#issuecomment-1773855755

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [pick](Branch2.0) fix row count unconsistent when join ordering (#24589) [doris]

2023-10-21 Thread via GitHub


xiaokang merged PR #25666:
URL: https://github.com/apache/doris/pull/25666


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch branch-2.0 updated: [fix](Nereids) fix row count unconsistent when join ordering (#24589) (#25666)

2023-10-21 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new f2f8a6afcae [fix](Nereids) fix row count unconsistent when join 
ordering (#24589) (#25666)
f2f8a6afcae is described below

commit f2f8a6afcae582fc0931efe9e5f61f271043b896
Author: 谢健 
AuthorDate: Sun Oct 22 00:45:52 2023 +0800

[fix](Nereids) fix row count unconsistent when join ordering (#24589) 
(#25666)

In the context of reorder join, when a new plan is generated, it may 
include a project operation. In this case, the newly generated join root and 
the original join root will no longer be in the same group. To avoid 
inconsistencies in the statistics between these two groups, we keep the child 
group's row count unchanged when the parent group expression is a project 
operation.
---
 .../nereids/jobs/cascades/DeriveStatsJob.java  | 12 +++
 .../nereids_tpcds_shape_sf100_p0/shape/query18.out | 14 ++--
 .../nereids_tpcds_shape_sf100_p0/shape/query59.out | 35 +
 .../nereids_tpcds_shape_sf100_p0/shape/query61.out | 36 +
 .../nereids_tpcds_shape_sf100_p0/shape/query71.out | 73 +-
 .../nereids_tpcds_shape_sf100_p0/shape/query72.out | 48 +++-
 .../nereids_tpcds_shape_sf100_p0/shape/query99.out |  8 +-
 .../data/nereids_tpch_shape_sf1000_p0/shape/q8.out | 49 ++--
 .../nereids_tpcds_shape_sf100_p0/rf/ds_rf18.groovy |  2 +-
 .../nereids_tpcds_shape_sf100_p0/rf/ds_rf59.groovy | 83 +++--
 .../nereids_tpcds_shape_sf100_p0/rf/ds_rf61.groovy | 86 +++---
 .../nereids_tpcds_shape_sf100_p0/rf/ds_rf72.groovy |  2 +-
 .../nereids_tpcds_shape_sf100_p0/rf/ds_rf99.groovy |  2 +-
 .../nereids_tpch_shape_sf1000_p0/rf/h_rf8.groovy   |  2 +-
 14 files changed, 261 insertions(+), 191 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/DeriveStatsJob.java
 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/DeriveStatsJob.java
index cfe952c0f26..ec65b9af14d 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/DeriveStatsJob.java
+++ 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/DeriveStatsJob.java
@@ -28,6 +28,7 @@ import org.apache.doris.nereids.metrics.consumer.LogConsumer;
 import org.apache.doris.nereids.metrics.event.StatsStateEvent;
 import org.apache.doris.nereids.stats.StatsCalculator;
 import org.apache.doris.nereids.trees.expressions.CTEId;
+import org.apache.doris.nereids.trees.plans.algebra.Project;
 import org.apache.doris.qe.ConnectContext;
 import org.apache.doris.statistics.Statistics;
 
@@ -116,6 +117,17 @@ public class DeriveStatsJob extends Job {
 
context.getCascadesContext().getConnectContext().getTotalHistogramMap()
 .putAll(statsCalculator.getTotalHistogramMap());
 }
+
+if (groupExpression.getPlan() instanceof Project) {
+// In the context of reorder join, when a new plan is 
generated, it may include a project operation.
+// In this case, the newly generated join root and the 
original join root will no longer be in the
+// same group. To avoid inconsistencies in the statistics 
between these two groups, we keep the
+// child group's row count unchanged when the parent group 
expression is a project operation.
+double parentRowCount = 
groupExpression.getOwnerGroup().getStatistics().getRowCount();
+groupExpression.children().forEach(g -> g.setStatistics(
+
g.getStatistics().withRowCountAndEnforceValid(parentRowCount))
+);
+}
 }
 }
 }
diff --git 
a/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query18.out 
b/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query18.out
index f7e24486fd6..2ce64b0b73c 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query18.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query18.out
@@ -10,15 +10,15 @@ PhysicalResultSink
 --hashAgg[LOCAL]
 PhysicalRepeat
 --PhysicalProject
-hashJoin[INNER_JOIN](customer.c_current_cdemo_sk = 
cd2.cd_demo_sk)
+hashJoin[INNER_JOIN](catalog_sales.cs_item_sk = 
item.i_item_sk)
+--PhysicalProject
+PhysicalOlapScan[item]
 --PhysicalDistribute
 PhysicalProject
---PhysicalOlapScan[customer_demographics]
---PhysicalDistribute
-PhysicalProject
---hashJoin[INNER_JOIN](catalog_sales.cs_item_sk = 
item.i_item_sk)
-PhysicalProject

Re: [PR] [pick](Branch2.0) generate left deep tree when stats is unknown [doris]

2023-10-21 Thread via GitHub


xiaokang merged PR #25702:
URL: https://github.com/apache/doris/pull/25702


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch branch-2.0 updated: [pick](Branch2.0) generate left deep tree when stats is unknown (#25702)

2023-10-21 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new ff786d05fb4 [pick](Branch2.0) generate left deep tree when stats is 
unknown (#25702)
ff786d05fb4 is described below

commit ff786d05fb4d05aee444a267883fb7998f8ea91f
Author: 谢健 
AuthorDate: Sun Oct 22 00:46:43 2023 +0800

[pick](Branch2.0) generate left deep tree when stats is unknown (#25702)
---
 .../org/apache/doris/nereids/cost/CostModelV1.java | 43 +-
 .../apache/doris/nereids/stats/JoinEstimation.java | 17 -
 .../trees/plans/physical/PhysicalHashJoin.java |  8 
 .../plans/physical/PhysicalNestedLoopJoin.java |  8 
 4 files changed, 74 insertions(+), 2 deletions(-)

diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java
index aa8f4d6cc7c..2aca6017b7b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java
@@ -22,6 +22,7 @@ import org.apache.doris.nereids.properties.DistributionSpec;
 import org.apache.doris.nereids.properties.DistributionSpecGather;
 import org.apache.doris.nereids.properties.DistributionSpecHash;
 import org.apache.doris.nereids.properties.DistributionSpecReplicated;
+import org.apache.doris.nereids.trees.expressions.Slot;
 import org.apache.doris.nereids.trees.plans.Plan;
 import org.apache.doris.nereids.trees.plans.physical.PhysicalAssertNumRows;
 import 
org.apache.doris.nereids.trees.plans.physical.PhysicalDeferMaterializeOlapScan;
@@ -311,17 +312,53 @@ class CostModelV1 extends PlanVisitor {
 }
 // TODO: since the outputs rows may expand a lot, penalty on it 
will cause bc never be chosen.
 // will refine this in next generation cost model.
+if (isStatsUnknown(physicalHashJoin, buildStats, probeStats)) {
+// forbid broadcast join when stats is unknown
+return CostV1.of(rightRowCount * buildSideFactor + 1 / 
leftRowCount,
+rightRowCount,
+0
+);
+}
 return CostV1.of(leftRowCount + rightRowCount * buildSideFactor + 
outputRowCount * probeSideFactor,
 rightRowCount,
 0
 );
 }
+if (isStatsUnknown(physicalHashJoin, buildStats, probeStats)) {
+return CostV1.of(rightRowCount + 1 / leftRowCount,
+rightRowCount,
+0);
+}
 return CostV1.of(leftRowCount + rightRowCount + outputRowCount,
 rightRowCount,
 0
 );
 }
 
+private boolean isStatsUnknown(PhysicalHashJoin join,
+Statistics build, Statistics probe) {
+for (Slot slot : join.getConditionSlot()) {
+if ((build.columnStatistics().containsKey(slot) && 
!build.columnStatistics().get(slot).isUnKnown)
+|| (probe.columnStatistics().containsKey(slot) && 
!probe.columnStatistics().get(slot).isUnKnown)) {
+continue;
+}
+return true;
+}
+return false;
+}
+
+private boolean isStatsUnknown(PhysicalNestedLoopJoin join,
+Statistics build, Statistics probe) {
+for (Slot slot : join.getConditionSlot()) {
+if ((build.columnStatistics().containsKey(slot) && 
!build.columnStatistics().get(slot).isUnKnown)
+|| (probe.columnStatistics().containsKey(slot) && 
!probe.columnStatistics().get(slot).isUnKnown)) {
+continue;
+}
+return true;
+}
+return false;
+}
+
 @Override
 public Cost visitPhysicalNestedLoopJoin(
 PhysicalNestedLoopJoin 
nestedLoopJoin,
@@ -330,7 +367,11 @@ class CostModelV1 extends PlanVisitor {
 Preconditions.checkState(context.arity() == 2);
 Statistics leftStatistics = context.getChildStatistics(0);
 Statistics rightStatistics = context.getChildStatistics(1);
-
+if (isStatsUnknown(nestedLoopJoin, leftStatistics, rightStatistics)) {
+return CostV1.of(rightStatistics.getRowCount() + 1 / 
leftStatistics.getRowCount(),
+rightStatistics.getRowCount(),
+0);
+}
 return CostV1.of(
 leftStatistics.getRowCount() * rightStatistics.getRowCount(),
 rightStatistics.getRowCount(),
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java 
b/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java
index ef4575e3308..0498d68d793 100644
--- 
a/fe/fe-core/src/main/java/org/apache/doris/

Re: [PR] [improvement](sync version) fe sync version with be [doris]

2023-10-21 Thread via GitHub


xiaokang merged PR #25718:
URL: https://github.com/apache/doris/pull/25718


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch branch-2.0 updated: [improvement](sync version) fe sync version with be (#25718)

2023-10-21 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 6fe9d5d1af9 [improvement](sync version) fe sync version with be 
(#25718)
6fe9d5d1af9 is described below

commit 6fe9d5d1af9b9d5475a04925aea5f9e365108255
Author: yujun 
AuthorDate: Sun Oct 22 00:47:56 2023 +0800

[improvement](sync version) fe sync version with be (#25718)
---
 be/src/http/action/debug_point_action.cpp  |  19 ++-
 be/src/util/debug_points.cpp   |  39 +++--
 be/src/util/debug_points.h |  72 -
 be/test/util/debug_points_test.cpp |  40 +
 .../java/org/apache/doris/catalog/Replica.java |  36 -
 .../apache/doris/catalog/TabletInvertedIndex.java  |  22 ++-
 .../org/apache/doris/clone/TabletSchedCtx.java |   7 +
 .../apache/doris/common/util/DebugPointUtil.java   |  95 +--
 .../apache/doris/httpv2/rest/DebugPointAction.java |  25 ++-
 .../java/org/apache/doris/master/MasterImpl.java   |   9 +-
 .../org/apache/doris/master/ReportHandler.java |  42 +++--
 .../org/apache/doris/clone/RepairVersionTest.java  | 177 +
 .../doris/common/util/DebugPointUtilTest.java  |  18 +++
 .../apache/doris/utframe/TestWithFeService.java|   2 +-
 14 files changed, 533 insertions(+), 70 deletions(-)

diff --git a/be/src/http/action/debug_point_action.cpp 
b/be/src/http/action/debug_point_action.cpp
index 08b1e116b2b..04aa38efaa4 100644
--- a/be/src/http/action/debug_point_action.cpp
+++ b/be/src/http/action/debug_point_action.cpp
@@ -21,6 +21,7 @@
 #include "http/http_channel.h"
 #include "http/http_status.h"
 #include "util/debug_points.h"
+#include "util/time.h"
 
 namespace doris {
 
@@ -43,17 +44,16 @@ void BaseDebugPointAction::handle(HttpRequest* req) {
 }
 
 Status AddDebugPointAction::_handle(HttpRequest* req) {
-std::string debug_point = req->param("debug_point");
+std::string name = req->param("debug_point");
 std::string execute = req->param("execute");
 std::string timeout = req->param("timeout");
-if (debug_point.empty()) {
+if (name.empty()) {
 return Status::InternalError("Empty debug point name");
 }
-int64_t execute_limit = -1;
-int64_t timeout_second = -1;
+auto debug_point = std::make_shared();
 try {
 if (!execute.empty()) {
-execute_limit = std::stol(execute);
+debug_point->execute_limit = std::stol(execute);
 }
 } catch (const std::exception& e) {
 return Status::InternalError("Invalid execute limit format, execute 
{}, err {}", execute,
@@ -61,14 +61,19 @@ Status AddDebugPointAction::_handle(HttpRequest* req) {
 }
 try {
 if (!timeout.empty()) {
-timeout_second = std::stol(timeout);
+int64_t timeout_second = std::stol(timeout);
+if (timeout_second > 0) {
+debug_point->expire_ms = MonotonicMillis() + timeout_second * 
MILLIS_PER_SEC;
+}
 }
 } catch (const std::exception& e) {
 return Status::InternalError("Invalid timeout format, timeout {}, err 
{}", timeout,
  e.what());
 }
 
-DebugPoints::instance()->add(debug_point, execute_limit, timeout_second);
+debug_point->params = *(req->params());
+
+DebugPoints::instance()->add(name, debug_point);
 
 return Status::OK();
 }
diff --git a/be/src/util/debug_points.cpp b/be/src/util/debug_points.cpp
index 587f8c944a3..43bb39df9a4 100644
--- a/be/src/util/debug_points.cpp
+++ b/be/src/util/debug_points.cpp
@@ -30,37 +30,42 @@ DebugPoints* DebugPoints::instance() {
 }
 
 bool DebugPoints::is_enable(const std::string& name) {
+return get_debug_point(name) != nullptr;
+}
+
+std::shared_ptr DebugPoints::get_debug_point(const std::string& 
name) {
 if (!config::enable_debug_points) {
-return false;
+return nullptr;
 }
 auto map_ptr = std::atomic_load_explicit(&_debug_points, 
std::memory_order_relaxed);
 auto it = map_ptr->find(name);
 if (it == map_ptr->end()) {
-return false;
+return nullptr;
 }
 
-auto& debug_point = *(it->second);
-if ((debug_point.expire_ms > 0 && MonotonicMillis() >= 
debug_point.expire_ms) ||
-(debug_point.execute_limit > 0 &&
- debug_point.execute_num.fetch_add(1, std::memory_order_relaxed) >=
- debug_point.execute_limit)) {
+auto debug_point = it->second;
+if ((debug_point->expire_ms > 0 && MonotonicMillis() >= 
debug_point->expire_ms) ||
+(debug_point->execute_limit > 0 &&
+ debug_point->execute_num.fetch_add(1, std::memory_order_relaxed) >=
+ debug_point->execute_limit)) {
 remove(name);
-return false;
+return nullptr;
 }
 
-

[doris] branch branch-2.0 updated: [feat](stats) Enable auto sample by default (#25607)

2023-10-21 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-2.0 by this push:
 new 41fe511ae8d [feat](stats) Enable auto sample by default (#25607)
41fe511ae8d is described below

commit 41fe511ae8db559b03baf42c6440c21edbfd8e41
Author: AKIRA <33112463+kikyou1...@users.noreply.github.com>
AuthorDate: Sun Oct 22 00:48:49 2023 +0800

[feat](stats) Enable auto sample by default (#25607)
---
 fe/fe-common/src/main/java/org/apache/doris/common/Config.java  | 2 +-
 .../src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java 
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 72ecebca781..8fc192a6d94 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -2164,7 +2164,7 @@ public class Config extends ConfigBase {
 @ConfField(description = 
{"是否开启大表自动sample,开启后对于大小超过huge_table_lower_bound_size_in_bytes会自动通过采样收集"
 + "统计信息", "Whether to enable automatic sampling for large tables, 
which, when enabled, automatically"
 + "collects statistics through sampling for tables larger than 
'huge_table_lower_bound_size_in_bytes'"})
-public static boolean enable_auto_sample = false;
+public static boolean enable_auto_sample = true;
 
 @ConfField(description = {
 "控制统计信息的自动触发作业执行记录的持久化行数",
diff --git 
a/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java 
b/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java
index 04b775bcd63..81cc97a63bf 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java
@@ -120,11 +120,11 @@ public abstract class BaseAnalysisTask {
 }
 
 protected void init(AnalysisInfo info) {
-tableSample = getTableSample();
 DBObjects dbObjects = 
StatisticsUtil.convertIdToObjects(info.catalogId, info.dbId, info.tblId);
 catalog = dbObjects.catalog;
 db = dbObjects.db;
 tbl = dbObjects.table;
+tableSample = getTableSample();
 // External Table level task doesn't contain a column. Don't need to 
do the column related analyze.
 if (info.externalTableLevelTask) {
 return;


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feat](stats) Enable auto sample by default [doris]

2023-10-21 Thread via GitHub


xiaokang merged PR #25607:
URL: https://github.com/apache/doris/pull/25607


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix](exec) Fix date dict infinite loop. [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #25570:
URL: https://github.com/apache/doris/pull/25570#issuecomment-1773858245

   TeamCity be ut coverage result:
Function Coverage: 37.04% (8301/22410) 
Line Coverage: 29.20% (66677/228330)
Region Coverage: 27.83% (34607/124360)
Branch Coverage: 24.42% (17583/71996)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/c797a35110db93fb57bb560c4790bd257d8375a7_c797a35110db93fb57bb560c4790bd257d8375a7/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix](exec) Fix date dict infinite loop. [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #25570:
URL: https://github.com/apache/doris/pull/25570#issuecomment-1773858496

   TeamCity be ut coverage result:
Function Coverage: 37.04% (8301/22410) 
Line Coverage: 29.20% (66675/228330)
Region Coverage: 27.83% (34612/124360)
Branch Coverage: 24.42% (17582/71996)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/c797a35110db93fb57bb560c4790bd257d8375a7_c797a35110db93fb57bb560c4790bd257d8375a7/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](alter backends)backend modify & drop & decommission by ids [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #25444:
URL: https://github.com/apache/doris/pull/25444#issuecomment-1773860036

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.21 seconds
stream load tsv:  557 seconds loaded 74807831229 Bytes, about 128 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  29.4 seconds inserted 1000 Rows, about 
340K ops/s
storage size: 17162340604 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [enhancement](recover) support skipping missing version in select by session variable [doris]

2023-10-21 Thread via GitHub


xy720 commented on PR #25654:
URL: https://github.com/apache/doris/pull/25654#issuecomment-1773861471

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [enhancement](recover) support skipping missing version in select by session variable [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25654:
URL: https://github.com/apache/doris/pull/25654#issuecomment-1773863530

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [enhancement](recover) support skipping missing version in select by session variable [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25654:
URL: https://github.com/apache/doris/pull/25654#issuecomment-1773867151

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](decimal) support decimal256 [doris]

2023-10-21 Thread via GitHub


jacktengg commented on PR #25386:
URL: https://github.com/apache/doris/pull/25386#issuecomment-1773870984

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](decimal) support decimal256 [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on code in PR #25386:
URL: https://github.com/apache/doris/pull/25386#discussion_r1367772481


##
be/test/vec/data_types/decimal_test.cpp:
##
@@ -0,0 +1,212 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include 
+#include 
+#include 
+
+#include 
+#include 
+
+#include "gtest/gtest_pred_impl.h"
+#include "runtime/raw_value.h"
+#include "runtime/type_limit.h"
+#include "util/string_parser.hpp"
+#include "vec/core/types.h"
+#include "vec/data_types/data_type_decimal.h"
+namespace doris::vectorized {
+
+TEST(DecimalTest, Decimal256) {
+// 

+Decimal256 dec1(type_limit::max());
+auto des_str = dec1.to_string(10);
+EXPECT_EQ(des_str,
+  
"99.99");
+des_str = dec1.to_string(0);
+EXPECT_EQ(des_str,
+  
"");
+des_str = dec1.to_string(76);
+EXPECT_EQ(des_str,
+  
"0.");
+
+auto dec2 = type_limit::min();
+des_str = dec2.to_string(10);
+EXPECT_EQ(des_str,
+  
"-99.99");
+des_str = dec2.to_string(0);
+EXPECT_EQ(des_str,
+  
"-");
+des_str = dec2.to_string(76);
+EXPECT_EQ(des_str,
+  
"-0.");
+
+// plus
+Decimal256 dec3 = dec1 + dec2;
+des_str = dec3.to_string(10);
+EXPECT_EQ(des_str, "0.00");
+des_str = dec3.to_string(0);
+EXPECT_EQ(des_str, "0");
+des_str = dec3.to_string(76);
+EXPECT_EQ(des_str,
+  
"0.");
+
+// minus
+dec2 = type_limit::max();
+dec3 = dec1 - dec2;
+des_str = dec3.to_string(10);
+EXPECT_EQ(des_str, "0.00");
+
+// multiply
+
+// divide
+dec1 = type_limit::max();
+dec2 = vectorized::Decimal256(10);
+dec3 = dec1 / dec2;
+des_str = dec3.to_string(1);
+EXPECT_EQ(des_str,
+  
"99.9");
+
+// overflow
+}
+
+TEST(DecimalTest, compare) {

Review Comment:
   warning: all parameters should be named in a function 
[readability-named-parameter]
   
   ```suggestion
   TEST(DecimalTest /*unused*/, compare /*unused*/) {
   ```
   



##
be/test/vec/data_types/decimal_test.cpp:
##
@@ -0,0 +1,212 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include 
+#include 
+#include 
+
+#include 
+#include 
+
+#include "gtest/gtest_pred_impl.h"
+#include "runtime/raw_value.h"
+#include "runtime/type_limit.h"
+#include "util/string_parser.hpp"
+#include "vec/core/types.h"
+#include "vec/data_types/data_type_decimal.h"
+namespace doris::vectorized {
+
+TEST(DecimalTest, Decimal256) {
+// 

+Decimal256 dec1(type_limit::max());
+auto des_str = dec1.to_string(10);
+EXPECT_EQ(des_str,
+  
"9

Re: [PR] [Fix](exec) Fix date dict infinite loop. [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #25570:
URL: https://github.com/apache/doris/pull/25570#issuecomment-1773877852

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.12 seconds
stream load tsv:  555 seconds loaded 74807831229 Bytes, about 128 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  29.2 seconds inserted 1000 Rows, about 
342K ops/s
storage size: 17161905632 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [enhancement](recover) support skipping missing version in select by session variable [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #25654:
URL: https://github.com/apache/doris/pull/25654#issuecomment-1773882417

   TeamCity be ut coverage result:
Function Coverage: 37.04% (8301/22411) 
Line Coverage: 29.20% (3/228334)
Region Coverage: 27.83% (34611/124366)
Branch Coverage: 24.41% (17578/72002)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/f096b41aaf47ad5f30fbb86a151b0ac7963d841f_f096b41aaf47ad5f30fbb86a151b0ac7963d841f/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](hive)Support hive tables after alter type. [doris]

2023-10-21 Thread via GitHub


hubgeter commented on PR #25138:
URL: https://github.com/apache/doris/pull/25138#issuecomment-1773890160

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](hive)Support hive tables after alter type. [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on code in PR #25138:
URL: https://github.com/apache/doris/pull/25138#discussion_r1367781725


##
be/src/vec/exec/format/parquet/vparquet_column_reader.cpp:
##
@@ -476,86 +481,109 @@ Status ScalarColumnReader::_try_load_dict_page(bool* 
loaded, bool* has_dict) {
 Status ScalarColumnReader::read_column_data(ColumnPtr& doris_column, 
DataTypePtr& type,
 ColumnSelectVector& select_vector, 
size_t batch_size,
 size_t* read_rows, bool* eof, bool 
is_dict_filter) {
-if (_chunk_reader->remaining_num_values() == 0) {
-if (!_chunk_reader->has_next_page()) {
-*eof = true;
-*read_rows = 0;
-return Status::OK();
-}
-RETURN_IF_ERROR(_chunk_reader->next_page());
-}
-if (_nested_column) {
-RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
-return _read_nested_column(doris_column, type, select_vector, 
batch_size, read_rows, eof,
-   is_dict_filter);
-}
-
-// generate the row ranges that should be read
-std::list read_ranges;
-_generate_read_ranges(_current_row_index,
-  _current_row_index + 
_chunk_reader->remaining_num_values(), read_ranges);
-if (read_ranges.size() == 0) {
-// skip the whole page
-_current_row_index += _chunk_reader->remaining_num_values();
-RETURN_IF_ERROR(_chunk_reader->skip_page());
-*read_rows = 0;
-} else {
-bool skip_whole_batch = false;
-// Determining whether to skip page or batch will increase the 
calculation time.
-// When the filtering effect is greater than 60%, it is possible to 
skip the page or batch.
-if (select_vector.has_filter() && select_vector.filter_ratio() > 0.6) {
-// lazy read
-size_t remaining_num_values = 0;
-for (auto& range : read_ranges) {
-remaining_num_values += range.last_row - range.first_row;
-}
-if (batch_size >= remaining_num_values &&
-select_vector.can_filter_all(remaining_num_values)) {
-// We can skip the whole page if the remaining values is 
filtered by predicate columns
-select_vector.skip(remaining_num_values);
-_current_row_index += _chunk_reader->remaining_num_values();
-RETURN_IF_ERROR(_chunk_reader->skip_page());
-*read_rows = remaining_num_values;
-if (!_chunk_reader->has_next_page()) {
-*eof = true;
-}
+bool need_convert = false;
+auto& parquet_physical_type = _chunk_meta.meta_data.type;
+auto& show_type = _field_schema->type.type;
+
+ColumnPtr src_column = ParquetConvert::get_column(parquet_physical_type, 
show_type,
+  doris_column, type, 
&need_convert);
+
+do {
+if (_chunk_reader->remaining_num_values() == 0) {
+if (!_chunk_reader->has_next_page()) {
+*eof = true;
+*read_rows = 0;
 return Status::OK();
 }
-skip_whole_batch =
-batch_size <= remaining_num_values && 
select_vector.can_filter_all(batch_size);
-if (skip_whole_batch) {
-select_vector.skip(batch_size);
-}
+RETURN_IF_ERROR(_chunk_reader->next_page());
 }
-// load page data to decode or skip values
-RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
-size_t has_read = 0;
-for (auto& range : read_ranges) {
-// generate the skipped values
-size_t skip_values = range.first_row - _current_row_index;
-RETURN_IF_ERROR(_skip_values(skip_values));
-_current_row_index += skip_values;
-// generate the read values
-size_t read_values =
-std::min((size_t)(range.last_row - range.first_row), 
batch_size - has_read);
-if (skip_whole_batch) {
-RETURN_IF_ERROR(_skip_values(read_values));
-} else {
-RETURN_IF_ERROR(_read_values(read_values, doris_column, type, 
select_vector,
- is_dict_filter));
+if (_nested_column) {
+RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent());
+RETURN_IF_ERROR(_read_nested_column(src_column, type, 
select_vector, batch_size,
+read_rows, eof, 
is_dict_filter));
+break;
+}
+
+// generate the row ranges that should be read
+std::list read_ranges;
+_generate_read_ranges(_current_row_index,
+  _current_row_index + 
_chunk_reader->remaining_num_values(),
+ 

Re: [PR] [feature](decimal) support decimal256 [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #25386:
URL: https://github.com/apache/doris/pull/25386#issuecomment-1773892872

   TeamCity be ut coverage result:
Function Coverage: 37.21% (8378/22513) 
Line Coverage: 29.37% (67536/229926)
Region Coverage: 27.95% (35028/125340)
Branch Coverage: 24.50% (17741/72414)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/4d81ec09ed4ccaf74c79c7e700cdb27c2c004b90_4d81ec09ed4ccaf74c79c7e700cdb27c2c004b90/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](decimal) support decimal256 [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #25386:
URL: https://github.com/apache/doris/pull/25386#issuecomment-1773901653

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.89 seconds
stream load tsv:  551 seconds loaded 74807831229 Bytes, about 129 
MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  29.0 seconds inserted 1000 Rows, about 
344K ops/s
storage size: 17162042504 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [fix](rf) fix minmax runtime filter [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25720:
URL: https://github.com/apache/doris/pull/25720#issuecomment-1773949947

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Fix] Fix the extension mysql_to_doris bug #18985 [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #18988:
URL: https://github.com/apache/doris/pull/18988#issuecomment-1773951098

   We're closing this PR because it hasn't been updated in a while.
   This isn't a judgement on the merit of the PR in any way. It's just a way of 
keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and feel free a maintainer 
to remove the Stale tag!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [enhancment](quantile_union): support quantile_union on nullable columns [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #17793:
URL: https://github.com/apache/doris/pull/17793#issuecomment-177395

   We're closing this PR because it hasn't been updated in a while.
   This isn't a judgement on the merit of the PR in any way. It's just a way of 
keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and feel free a maintainer 
to remove the Stale tag!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



[doris] branch branch-2.0 updated (41fe511ae8d -> 243a5f52cdf)

2023-10-21 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a change to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git


from 41fe511ae8d [feat](stats) Enable auto sample by default (#25607)
 new bd3ee3ebdca [feature](function) Support SHA family functions (#24342)
 new ef099419375 [Improvement](hive-udf)(doc) minimize hive-udf and add 
some docs. (#24786)
 new a92632700cd [Enhance](regression) add hive out file check (#25475)
 new 3dab722500f [cases](regresstest)add json format regress test for 
nested types (#25397)
 new 243a5f52cdf [Improvement](statistics)Collect stats for hive partition 
column using metadata (#24853)

The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 be/src/util/sha.cpp| 118 ++
 be/src/util/sha.h  |  75 ++
 be/src/vec/functions/function_string.cpp   |  11 +-
 be/src/vec/functions/function_string.h | 107 -
 docs/en/docs/ecosystem/hive-bitmap-udf.md  |  56 -
 docs/en/docs/query-acceleration/statistics.md  |  15 +-
 .../sql-functions/encrypt-digest-functions/sha.md  |  53 +
 .../sql-functions/encrypt-digest-functions/sha2.md |  70 ++
 docs/sidebars.json |   4 +-
 docs/zh-CN/docs/ecosystem/hive-bitmap-udf.md   |  56 -
 docs/zh-CN/docs/query-acceleration/statistics.md   | 184 +--
 .../sql-functions/encrypt-digest-functions/sha.md  |  54 +
 .../sql-functions/encrypt-digest-functions/sha2.md |  70 ++
 fe/fe-core/src/main/cup/sql_parser.cup |  20 ++
 .../apache/doris/analysis/AnalyzeProperties.java   |   7 +-
 .../org/apache/doris/analysis/AnalyzeStmt.java |   4 +
 .../apache/doris/analysis/FunctionCallExpr.java|  13 ++
 .../doris/catalog/BuiltinScalarFunctions.java  |   4 +
 .../doris/datasource/hive/HiveMetaStoreCache.java  |  14 +-
 .../apache/doris/external/hive/util/HiveUtil.java  |  16 ++
 .../trees/expressions/functions/scalar/Sha1.java   |  68 ++
 .../trees/expressions/functions/scalar/Sha2.java   |  69 ++
 .../expressions/visitor/ScalarFunctionVisitor.java |  10 +
 .../org/apache/doris/statistics/AnalysisInfo.java  |   8 +-
 .../doris/statistics/AnalysisInfoBuilder.java  |  20 +-
 .../apache/doris/statistics/AnalysisManager.java   |   1 +
 .../apache/doris/statistics/BaseAnalysisTask.java  |  18 ++
 .../apache/doris/statistics/HMSAnalysisTask.java   | 154 +++--
 .../doris/statistics/util/StatisticsUtil.java  |   5 +
 fe/fe-core/src/main/jflex/sql_scanner.flex |   1 +
 fe/hive-udf/pom.xml|  75 --
 gensrc/script/doris_builtins_functions.py  |   7 +
 .../test_nestedtypes_json_insert_into_with_s3.out  |  18 +-
 .../hive/test_prepare_hive_data_in_case.out|  11 +
 .../encryption_digest/test_digest.out  |  39 
 .../sql_functions/encryption_digest/test_md5.out   |   6 -
 .../org/apache/doris/regression/suite/Suite.groovy |  48 +++-
 ...est_nestedtypes_json_insert_into_with_s3.groovy | 253 +
 .../hive/test_prepare_hive_data_in_case.groovy |  11 +-
 .../hive/test_hive_partition_column_analyze.groovy | 230 +++
 .../{test_md5.groovy => test_digest.groovy}|  19 +-
 41 files changed, 1737 insertions(+), 285 deletions(-)
 create mode 100644 be/src/util/sha.cpp
 create mode 100644 be/src/util/sha.h
 create mode 100644 
docs/en/docs/sql-manual/sql-functions/encrypt-digest-functions/sha.md
 create mode 100644 
docs/en/docs/sql-manual/sql-functions/encrypt-digest-functions/sha2.md
 create mode 100644 
docs/zh-CN/docs/sql-manual/sql-functions/encrypt-digest-functions/sha.md
 create mode 100644 
docs/zh-CN/docs/sql-manual/sql-functions/encrypt-digest-functions/sha2.md
 create mode 100644 
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Sha1.java
 create mode 100644 
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Sha2.java
 create mode 100644 
regression-test/data/external_table_p0/hive/test_prepare_hive_data_in_case.out
 create mode 100644 
regression-test/data/query_p0/sql_functions/encryption_digest/test_digest.out
 delete mode 100644 
regression-test/data/query_p0/sql_functions/encryption_digest/test_md5.out
 create mode 100644 
regression-test/suites/datatype_p0/nested_types/query/test_nestedtypes_json_insert_into_with_s3.groovy
 create mode 100644 
regression-test/suites/external_table_p2/hive/test_hive_partition_column_analyze.groovy
 rename 
regression-test/suites/query_p0/sql_functions/encryption_digest/{test_md5.groovy
 => test_digest.groovy} (53%)


---

[doris] 04/05: [cases](regresstest)add json format regress test for nested types (#25397)

2023-10-21 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 3dab722500f3062e0938d88d019736cd579bbe10
Author: amory 
AuthorDate: Tue Oct 17 10:16:52 2023 +0800

[cases](regresstest)add json format regress test for nested types (#25397)
---
 .../test_nestedtypes_json_insert_into_with_s3.out  |  18 +-
 ...est_nestedtypes_json_insert_into_with_s3.groovy | 253 +
 2 files changed, 262 insertions(+), 9 deletions(-)

diff --git 
a/regression-test/data/datatype_p0/nested_types/query/test_nestedtypes_json_insert_into_with_s3.out
 
b/regression-test/data/datatype_p0/nested_types/query/test_nestedtypes_json_insert_into_with_s3.out
index 21fb51ac176..a125a032e3c 100644
--- 
a/regression-test/data/datatype_p0/nested_types/query/test_nestedtypes_json_insert_into_with_s3.out
+++ 
b/regression-test/data/datatype_p0/nested_types/query/test_nestedtypes_json_insert_into_with_s3.out
@@ -1,33 +1,33 @@
 -- This file is automatically generated. You should know what you did if you 
want to edit this
 -- !sql_arr_json_without_quote_s3 --
-0  
[false,true,false,true,true,true,false,false,false,false,false,true,true,true,false,false,true,true,true,false,false,false,false,true,false,true,true,false,true,false,true,false,true,false,true,true,false,false,true,true,false,false,true,true,false,true,false,true,true,false,false,true,true,false,false,true,true,true,true,false,false,false,false,false,false,false,true,true,true,true,false,true,false,true,true,true,true,false,false,false,true,true,true,false,true,false,false,false,false
 [...]
+0  
[false,true,false,true,true,true,false,false,false,false,false,true,true,true,false,false,true,true,true,false,false,false,false,true,false,true,true,false,true,false,true,false,true,false,true,true,false,false,true,true,false,false,true,true,false,true,false,true,true,false,false,true,true,false,false,true,true,true,true,false,false,false,false,false,false,false,true,true,true,true,false,true,false,true,true,true,true,false,false,false,true,true,true,false,true,false,false,false,false
 [...]
 
 -- !sql_arr_json_without_quote_doris --
-0  [0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 
1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 
0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 
1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1]
[NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 
NULL, NULL, NULL, NULL, NULL, NU [...]
+0  [0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 
1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 
0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 
1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1]
[NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 
NULL, NULL, NULL, NULL, NULL, NU [...]
 
 -- !sql_arr_json_without_quote_s3 --
-0  
[[false,true],[true,true],[false,false],[],[false],[false],[],[],[],[false,false],[],[],[true],[true],[false],[true],[true],[],[true,false],[],[],[false],[false],[],[true,false],[false,true],[],[false,false],[true,false],[false,true],[false,false],[],[],[true],[true],[true],[],[],[false,true],[false,false],[],[],[true,false],[false],[false],[true],[],[],[],[true],[false,false],[true,false],[false,false],[false],[false,true],[],[false],[false,true],[false,false],[],[false],[true],[],[],
 [...]
+0  
[[false,true],[true,true],[false,false],[],[false],[false],[],[],[],[false,false],[],[],[true],[true],[false],[true],[true],[],[true,false],[],[],[false],[false],[],[true,false],[false,true],[],[false,false],[true,false],[false,true],[false,false],[],[],[true],[true],[true],[],[],[false,true],[false,false],[],[],[true,false],[false],[false],[true],[],[],[],[true],[false,false],[true,false],[false,false],[false],[false,true],[],[false],[false,true],[false,false],[],[false],[true],[],[],
 [...]
 
 -- !sql_arr_json_without_quote_doris --
-0  [[0, 1], [1, 1], [0, 0], [], [0], [0], [], [], [], [0, 0], [], [], [1], 
[1], [0], [1], [1], [], [1, 0], [], [], [0], [0], [], [1, 0], [0, 1], [], [0, 
0], [1, 0], [0, 1], [0, 0], [], [], [1], [1], [1], [], [], [0, 1], [0, 0], [], 
[], [1, 0], [0], [0], [1], [], [], [], [1], [0, 0], [1, 0], [0, 0], [0], [0, 
1], [], [0], [0, 1], [0, 0], [], [0], [1], [], [], [], [1, 0], [], [], [1], 
[1], [], [], [1, 0], [1, 0], [0, 0], [], [], [0, 0], [0, 1], [1], [1], [1], [1, 
0], [1], [], [0, 0], [], [0, [...]
+0  [[0, 1], [1, 1], [0, 0], [], [0], [0], [], [], [], [0, 0], [], [], [1], 
[1], [0], [1], [1], [], [1, 0], [], [], [0], [0], [],

[doris] 01/05: [feature](function) Support SHA family functions (#24342)

2023-10-21 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git

commit bd3ee3ebdca08743c14ac62f971929f7d98aee68
Author: zclllyybb 
AuthorDate: Wed Sep 20 17:21:45 2023 +0800

[feature](function) Support SHA family functions (#24342)
---
 be/src/util/sha.cpp| 118 +
 be/src/util/sha.h  |  75 +
 be/src/vec/functions/function_string.cpp   |  11 +-
 be/src/vec/functions/function_string.h | 107 ++-
 .../sql-functions/encrypt-digest-functions/sha.md  |  53 +
 .../sql-functions/encrypt-digest-functions/sha2.md |  70 
 docs/sidebars.json |   4 +-
 .../sql-functions/encrypt-digest-functions/sha.md  |  54 ++
 .../sql-functions/encrypt-digest-functions/sha2.md |  70 
 .../apache/doris/analysis/FunctionCallExpr.java|  13 +++
 .../doris/catalog/BuiltinScalarFunctions.java  |   4 +
 .../trees/expressions/functions/scalar/Sha1.java   |  68 
 .../trees/expressions/functions/scalar/Sha2.java   |  69 
 .../expressions/visitor/ScalarFunctionVisitor.java |  10 ++
 gensrc/script/doris_builtins_functions.py  |   7 ++
 .../encryption_digest/test_digest.out  |  39 +++
 .../sql_functions/encryption_digest/test_md5.out   |   6 --
 .../{test_md5.groovy => test_digest.groovy}|  19 +++-
 18 files changed, 781 insertions(+), 16 deletions(-)

diff --git a/be/src/util/sha.cpp b/be/src/util/sha.cpp
new file mode 100644
index 000..68099ff9269
--- /dev/null
+++ b/be/src/util/sha.cpp
@@ -0,0 +1,118 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "util/sha.h"
+
+#include 
+
+#include 
+
+namespace doris {
+
+constexpr static char dig_vec_lower[] = "0123456789abcdef";
+
+void SHA1Digest::reset(const void* data, size_t length) {
+SHA1_Init(&_sha_ctx);
+SHA1_Update(&_sha_ctx, data, length);
+}
+
+std::string_view SHA1Digest::digest() {
+unsigned char buf[SHA_DIGEST_LENGTH];
+SHA1_Final(buf, &_sha_ctx);
+
+char* to = _reuse_hex;
+for (int i = 0; i < SHA_DIGEST_LENGTH; ++i) {
+*to++ = dig_vec_lower[buf[i] >> 4];
+*to++ = dig_vec_lower[buf[i] & 0x0F];
+}
+
+return std::string_view {_reuse_hex, _reuse_hex + 2 * SHA_DIGEST_LENGTH};
+}
+
+void SHA224Digest::reset(const void* data, size_t length) {
+SHA224_Init(&_sha224_ctx);
+SHA224_Update(&_sha224_ctx, data, length);
+}
+
+std::string_view SHA224Digest::digest() {
+unsigned char buf[SHA224_DIGEST_LENGTH];
+SHA224_Final(buf, &_sha224_ctx);
+
+char* to = _reuse_hex;
+for (int i = 0; i < SHA224_DIGEST_LENGTH; ++i) {
+*to++ = dig_vec_lower[buf[i] >> 4];
+*to++ = dig_vec_lower[buf[i] & 0x0F];
+}
+
+return std::string_view {_reuse_hex, _reuse_hex + 2 * 
SHA224_DIGEST_LENGTH};
+}
+
+void SHA256Digest::reset(const void* data, size_t length) {
+SHA256_Init(&_sha256_ctx);
+SHA256_Update(&_sha256_ctx, data, length);
+}
+
+std::string_view SHA256Digest::digest() {
+unsigned char buf[SHA256_DIGEST_LENGTH];
+SHA256_Final(buf, &_sha256_ctx);
+
+char* to = _reuse_hex;
+for (int i = 0; i < SHA256_DIGEST_LENGTH; ++i) {
+*to++ = dig_vec_lower[buf[i] >> 4];
+*to++ = dig_vec_lower[buf[i] & 0x0F];
+}
+
+return std::string_view {_reuse_hex, _reuse_hex + 2 * 
SHA256_DIGEST_LENGTH};
+}
+
+void SHA384Digest::reset(const void* data, size_t length) {
+SHA384_Init(&_sha384_ctx);
+SHA384_Update(&_sha384_ctx, data, length);
+}
+
+std::string_view SHA384Digest::digest() {
+unsigned char buf[SHA384_DIGEST_LENGTH];
+SHA384_Final(buf, &_sha384_ctx);
+
+char* to = _reuse_hex;
+for (int i = 0; i < SHA384_DIGEST_LENGTH; ++i) {
+*to++ = dig_vec_lower[buf[i] >> 4];
+*to++ = dig_vec_lower[buf[i] & 0x0F];
+}
+
+return std::string_view {_reuse_hex, _reuse_hex + 2 * 
SHA384_DIGEST_LENGTH};
+}
+
+void SHA512Digest::reset(const void* data, size_t length) {
+SHA512_Init(&_sha512_ctx);
+SHA512_Update(&

[doris] 03/05: [Enhance](regression) add hive out file check (#25475)

2023-10-21 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git

commit a92632700cdd32481e8acbf7ca4f81e7cdc922a6
Author: zhangguoqiang <18372634...@163.com>
AuthorDate: Tue Oct 17 10:11:57 2023 +0800

[Enhance](regression) add hive out file check (#25475)

add hive out file check
fix hive sql state with " ; "
---
 .../hive/test_prepare_hive_data_in_case.out| 11 +
 .../org/apache/doris/regression/suite/Suite.groovy | 48 +++---
 .../hive/test_prepare_hive_data_in_case.groovy | 11 -
 3 files changed, 62 insertions(+), 8 deletions(-)

diff --git 
a/regression-test/data/external_table_p0/hive/test_prepare_hive_data_in_case.out
 
b/regression-test/data/external_table_p0/hive/test_prepare_hive_data_in_case.out
new file mode 100644
index 000..7259a57a915
--- /dev/null
+++ 
b/regression-test/data/external_table_p0/hive/test_prepare_hive_data_in_case.out
@@ -0,0 +1,11 @@
+-- This file is automatically generated. You should know what you did if you 
want to edit this
+-- !hive_docker_01 --
+eeefff
+cccddd
+aaabbb
+
+-- !sql_02 --
+eeefff
+cccddd
+aaabbb
+
diff --git 
a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
 
b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
index 6cd22bb663f..f03a59a0042 100644
--- 
a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
+++ 
b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy
@@ -551,7 +551,7 @@ class Suite implements GroovyInterceptable {
 }
 
 List> hive_docker(String sqlStr, boolean isOrder = false){
-String cleanedSqlStr = sqlStr.replaceAll(/;+$/, '')
+String cleanedSqlStr = sqlStr.replaceAll("\\s*;\\s*\$", "")
 def (result, meta) = 
JdbcUtils.executeToList(context.getHiveDockerConnection(), cleanedSqlStr)
 if (isOrder) {
 result = DataUtils.sortByToString(result)
@@ -560,7 +560,7 @@ class Suite implements GroovyInterceptable {
 }
 
 List> hive_remote(String sqlStr, boolean isOrder = false){
-String cleanedSqlStr = sqlStr.replaceAll(/;+$/, '')
+String cleanedSqlStr = sqlStr.replaceAll("\\s*;\\s*\$", "")
 def (result, meta) = 
JdbcUtils.executeToList(context.getHiveRemoteConnection(), cleanedSqlStr)
 if (isOrder) {
 result = DataUtils.sortByToString(result)
@@ -572,9 +572,23 @@ class Suite implements GroovyInterceptable {
 if (context.config.generateOutputFile || 
context.config.forceGenerateOutputFile) {
 Tuple2>, ResultSetMetaData> tupleResult = null
 if (arg instanceof PreparedStatement) {
-tupleResult = 
JdbcUtils.executeToStringList(context.getConnection(),  (PreparedStatement) arg)
+if (tag.contains("hive_docker")) {
+tupleResult = 
JdbcUtils.executeToStringList(context.getHiveDockerConnection(),  
(PreparedStatement) arg)
+}else if (tag.contains("hive_remote")) {
+tupleResult = 
JdbcUtils.executeToStringList(context.getHiveRemoteConnection(),  
(PreparedStatement) arg)
+}
+else{
+tupleResult = 
JdbcUtils.executeToStringList(context.getConnection(),  (PreparedStatement) arg)
+}
 } else {
-tupleResult = 
JdbcUtils.executeToStringList(context.getConnection(),  (String) arg)
+if (tag.contains("hive_docker")) {
+tupleResult = 
JdbcUtils.executeToStringList(context.getHiveDockerConnection(), (String) arg)
+}else if (tag.contains("hive_remote")) {
+tupleResult = 
JdbcUtils.executeToStringList(context.getHiveRemoteConnection(), (String) arg)
+}
+else{
+tupleResult = 
JdbcUtils.executeToStringList(context.getConnection(),  (String) arg)
+}
 }
 def (result, meta) = tupleResult
 if (isOrder) {
@@ -596,9 +610,23 @@ class Suite implements GroovyInterceptable {
 OutputUtils.TagBlockIterator expectCsvResults = 
context.getOutputIterator().next()
 Tuple2>, ResultSetMetaData> tupleResult = null
 if (arg instanceof PreparedStatement) {
-tupleResult = 
JdbcUtils.executeToStringList(context.getConnection(),  (PreparedStatement) arg)
+if (tag.contains("hive_docker")) {
+tupleResult = 
JdbcUtils.executeToStringList(context.getHiveDockerConnection(),  
(PreparedStatement) arg)
+}else if (tag.contains("hive_remote")) {
+tupleResult = 
JdbcUtils.executeToStringList(context.getHiveRemoteConnection(),  
(PreparedStatement) arg)
+   

[doris] 05/05: [Improvement](statistics)Collect stats for hive partition column using metadata (#24853)

2023-10-21 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git

commit 243a5f52cdf8698d72aac242fb2150288ab52185
Author: Jibing-Li <64681310+jibing...@users.noreply.github.com>
AuthorDate: Tue Oct 17 10:31:57 2023 +0800

[Improvement](statistics)Collect stats for hive partition column using 
metadata (#24853)

Hive partition columns' stats could be calculated from hive metastore data. 
Doesn't need to execute sql to get the stats.
This PR is using hive partition metadata to collect partition column stats.
---
 docs/en/docs/query-acceleration/statistics.md  |  15 +-
 docs/zh-CN/docs/query-acceleration/statistics.md   | 184 +
 fe/fe-core/src/main/cup/sql_parser.cup |  20 ++
 .../apache/doris/analysis/AnalyzeProperties.java   |   7 +-
 .../org/apache/doris/analysis/AnalyzeStmt.java |   4 +
 .../doris/datasource/hive/HiveMetaStoreCache.java  |  14 +-
 .../apache/doris/external/hive/util/HiveUtil.java  |  16 ++
 .../org/apache/doris/statistics/AnalysisInfo.java  |   8 +-
 .../doris/statistics/AnalysisInfoBuilder.java  |  20 +-
 .../apache/doris/statistics/AnalysisManager.java   |   1 +
 .../apache/doris/statistics/BaseAnalysisTask.java  |  18 ++
 .../apache/doris/statistics/HMSAnalysisTask.java   | 154 +++---
 .../doris/statistics/util/StatisticsUtil.java  |   5 +
 fe/fe-core/src/main/jflex/sql_scanner.flex |   1 +
 .../hive/test_hive_partition_column_analyze.groovy | 230 +
 15 files changed, 469 insertions(+), 228 deletions(-)

diff --git a/docs/en/docs/query-acceleration/statistics.md 
b/docs/en/docs/query-acceleration/statistics.md
index 2ab48ebc573..28795d01deb 100644
--- a/docs/en/docs/query-acceleration/statistics.md
+++ b/docs/en/docs/query-acceleration/statistics.md
@@ -79,8 +79,11 @@ The user triggers a manual collection job through a 
statement `ANALYZE` to colle
 Column statistics collection syntax:
 
 ```SQL
-ANALYZE TABLE | DATABASE table_name | db_name
-[ (column_name [, ...]) ][ [ WITH SYNC ] [ WITH INCREMENTAL ] [ WITH 
SAMPLE PERCENT | ROWS ] [ WITH PERIOD ] [WITH HISTOGRAM]][ PROPERTIES 
("key" = "value", ...) ];
+ANALYZE < TABLE | DATABASE table_name | db_name >
+[ PARTITIONS [(*) | (partition_name [, ...]) | WITH RECENT COUNT ] ]
+[ (column_name [, ...]) ]
+[ [ WITH SYNC ] [ WITH SAMPLE PERCENT | ROWS ] [ WITH SQL ] ]
+[ PROPERTIES ("key" = "value", ...) ];
 ```
 
 Explanation:
@@ -121,8 +124,12 @@ mysql -uroot -P9030 -h192.168.xxx.xxx```
 
 Create a data table:
 
-```SQL
-mysql> CREATE DATABASE IF NOT EXISTS stats_test;
+- `table_name`: Specifies the target table. It can be in the 
`db_name.table_name` format.
+- `partition_name`: The specified target partitions(for hive external table 
only)。Must be partitions exist in `table_name`. Multiple partition names are 
separated by commas. e.g. for single level partition: 
PARTITIONS(`event_date=20230706`), for multi level partition: 
PARTITIONS(`nation=US/city=Washington`). PARTITIONS(*) specifies all 
partitions, PARTITIONS WITH RECENT 30 specifies the latest 30 partitions.
+- `column_name`: Specifies the target column. It must be an existing column in 
`table_name`, and multiple column names are separated by commas.
+- `sync`: Collect statistics synchronously. Returns upon completion. If not 
specified, it executes asynchronously and returns a task ID.
+- `sample percent | rows`: Collect statistics using sampling. You can specify 
either the sampling percentage or the number of sampled rows.
+- `sql`: Collect statistics for external partition column with sql. By 
default, it uses meta data for partition columns, which is faster but may 
inaccurate for row count and size. Using sql could collect the accurate stats.
 
 mysql> CREATE TABLE IF NOT EXISTS stats_test.example_tbl (
 `user_id` LARGEINT NOT NULL,`date` DATEV2 NOT NULL,
`city` VARCHAR(20),`age` SMALLINT,`sex` TINYINT,
`last_visit_date` DATETIME REPLACE,`cost` BIGINT SUM,
`max_dwell_time` INT MAX,`min_dwell_time` INT MIN) ENGINE=OLAP
AGGREGATE KEY(`user_id`, `date`, `city`, `age`, `sex`)PARTITION BY 
LIST(`date`)(PARTITION `p_201701` VALUES IN ("2017-10-01"),
PARTITION `p_201702` VALUES IN ("2017-10-02"),   [...]
diff --git a/docs/zh-CN/docs/query-acceleration/statistics.md 
b/docs/zh-CN/docs/query-acceleration/statistics.md
index 2d893a39fc1..0362f989c1e 100644
--- a/docs/zh-CN/docs/query-acceleration/statistics.md
+++ b/docs/zh-CN/docs/query-acceleration/statistics.md
@@ -79,9 +79,10 @@ Doris 查询优化器使用统计信息来确定查询最有效的执行计划
 列统计信息收集语法:
 
 ```SQL
-ANALYZE TABLE | DATABASE table_name | db_name
+ANALYZE < TABLE | DATABASE table_name | db_name > 
+[ PARTITIONS [(*) | (partition_name [, ...]) | WITH RECENT COUNT ] ]
 [ (column_name [, ...]) ]
-[ [ WITH SYN

[doris] 02/05: [Improvement](hive-udf)(doc) minimize hive-udf and add some docs. (#24786)

2023-10-21 Thread kxiao
This is an automated email from the ASF dual-hosted git repository.

kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git

commit ef0994193758523aec0eb3398e016ec9b2bea7c6
Author: yagagagaga 
AuthorDate: Mon Oct 16 16:47:21 2023 +0800

[Improvement](hive-udf)(doc) minimize hive-udf and add some docs. (#24786)
---
 docs/en/docs/ecosystem/hive-bitmap-udf.md| 56 -
 docs/zh-CN/docs/ecosystem/hive-bitmap-udf.md | 56 -
 fe/hive-udf/pom.xml  | 75 
 3 files changed, 163 insertions(+), 24 deletions(-)

diff --git a/docs/en/docs/ecosystem/hive-bitmap-udf.md 
b/docs/en/docs/ecosystem/hive-bitmap-udf.md
index 6629b9d6a24..7bb93bc90ba 100644
--- a/docs/en/docs/ecosystem/hive-bitmap-udf.md
+++ b/docs/en/docs/ecosystem/hive-bitmap-udf.md
@@ -72,7 +72,7 @@ mvn package -Dmaven.test.skip=true
 --You can also just package the hive-udf module
 mvn package -pl hive-udf -am -Dmaven.test.skip=true
 ```
-After packaging and compiling, enter the hive-udf directory and there will be 
a target directory,There will be hive-udf-jar-with-dependencies.jar package
+After packaging and compiling, enter the hive-udf directory and there will be 
a target directory,There will be hive-udf.jar package
 
 ```sql
 -- Load the Hive Bitmap Udf jar package (Upload the compiled hive-udf jar 
package to HDFS)
@@ -108,4 +108,58 @@ select k1,bitmap_union(uuid) from hive_bitmap_table group 
by k1
 
 ## Hive Bitmap import into Doris
 
+
+
+### Method 1:Catalog (recommended)
+
+
+
+When create a Hive table in the format specified as TEXT, for Binary type, 
Hive will be saved as a bash64 encoded string. Therefore, the binary data can 
be directly saved as Bitmap through bitmap_from_base64 function by using  
Doris's Hive Catalog.
+
+Here is a full example:
+
+1. Creating Hive Tables in Hive
+
+```sql
+CREATE TABLE IF NOT EXISTS `test`.`hive_bitmap_table`(
+`k1`   int   COMMENT '',
+`k2`   StringCOMMENT '',
+`k3`   StringCOMMENT '',
+`uuid` binaryCOMMENT 'bitmap'
+) stored as textfile 
+```
+
+2. [Creating a Catalog in Doris](../lakehouse/multi-catalog/hive)
+
+```sql
+CREATE CATALOG hive PROPERTIES (
+'type'='hms',
+'hive.metastore.uris' = 'thrift://127.0.0.1:9083'
+);
+```
+
+3. Create Doris internal table
+
+```sql
+CREATE TABLE IF NOT EXISTS `test`.`doris_bitmap_table`(
+`k1`   int   COMMENT '',
+`k2`   StringCOMMENT '',
+`k3`   StringCOMMENT '',
+`uuid` BITMAP  BITMAP_UNION  COMMENT 'bitmap'
+)
+AGGREGATE KEY(k1, k2, k3)
+DISTRIBUTED BY HASH(`user_id`) BUCKETS 1
+PROPERTIES (
+"replication_allocation" = "tag.location.default: 1"
+);
+```
+
+4. Inserting data from Hive into Doris
+
+```sql
+insert into doris_bitmap_table select k1, k2, k3, bitmap_from_base64(uuid) 
from hive.test.hive_bitmap_table;
+```
+
+### Method 2:Spark Load
+
  see details: [Spark 
Load](../data-operate/import/import-way/spark-load-manual.md) -> Basic 
operation -> Create load(Example 3: when the upstream data source is hive 
binary type table)
diff --git a/docs/zh-CN/docs/ecosystem/hive-bitmap-udf.md 
b/docs/zh-CN/docs/ecosystem/hive-bitmap-udf.md
index 7f886352ed4..d10a9250283 100644
--- a/docs/zh-CN/docs/ecosystem/hive-bitmap-udf.md
+++ b/docs/zh-CN/docs/ecosystem/hive-bitmap-udf.md
@@ -74,7 +74,7 @@ mvn package -Dmaven.test.skip=true
 --也可以只打hive-udf module
 mvn package -pl hive-udf -am -Dmaven.test.skip=true
 ```
-打包编译完成进入hive-udf目录会有target目录,里面就会有打包完成的hive-udf-jar-with-dependencies.jar包
+打包编译完成进入hive-udf目录会有target目录,里面就会有打包完成的hive-udf.jar包
 
 ```sql
 
@@ -117,4 +117,58 @@ select k1,bitmap_union(uuid) from hive_bitmap_table group 
by k1
 
 ## Hive bitmap 导入 doris
 
+
+
+### 方法一:Catalog (推荐)
+
+
+
+创建 Hive 表指定为 TEXT 格式,此时,对于 Binary 类型,Hive 会以 bash64 编码的字符串形式保存,此时可以通过 Hive 
Catalog 的形式,直接将位图数据通过 bitmap_from_bash64 函数插入到 Doris 内部。
+
+以下是一个完整的例子:
+
+1. 在 Hive 中创建 Hive 表
+
+```sql
+CREATE TABLE IF NOT EXISTS `test`.`hive_bitmap_table`(
+`k1`   int   COMMENT '',
+`k2`   StringCOMMENT '',
+`k3`   StringCOMMENT '',
+`uuid` binaryCOMMENT 'bitmap'
+) stored as textfile 
+```
+
+2. [在 Doris 中创建 Catalog](../lakehouse/multi-catalog/hive)
+
+```sql
+CREATE CATALOG hive PROPERTIES (
+'type'='hms',
+'hive.metastore.uris' = 'thrift://127.0.0.1:9083'
+);
+```
+
+3. 创建 Doris 内表
+
+```sql
+CREATE TABLE IF NOT EXISTS `test`.`doris_bitmap_table`(
+`k1`   int   COMMENT '',
+`k2`   StringCOMMENT '',
+`k3`   StringCOMMENT '',
+`uuid` BITMAP  BITMAP_UNION  COMMENT 'bitmap'
+)
+AGGREGATE KEY(k1, k2, k3)
+DISTRIBUTED BY HASH(`user_id`) BUCKETS 1
+PROPERTIES (
+"replication_allocation" = "tag.location.default: 1"
+);
+```
+
+4. 从 Hive 插入数据到 Doris 中
+
+```sql
+insert into doris_bitmap_table select k1, k2, k3, bitmap_from_base64(uuid) 
from hive.test.hive_bitmap_table;
+```
+
+##

Re: [PR] [feature](alter backends)backend modify & drop & decommission by ids [doris]

2023-10-21 Thread via GitHub


liugddx commented on PR #25444:
URL: https://github.com/apache/doris/pull/25444#issuecomment-1773982701

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](CANCEL-ALTER-SYSTEM)decommission backend by ids [doris]

2023-10-21 Thread via GitHub


liugddx commented on PR #25441:
URL: https://github.com/apache/doris/pull/25441#issuecomment-1773983098

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](decimal) support decimal256 [doris]

2023-10-21 Thread via GitHub


jacktengg commented on PR #25386:
URL: https://github.com/apache/doris/pull/25386#issuecomment-1773983159

   run p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](alter backends)backend modify & drop & decommission by ids [doris]

2023-10-21 Thread via GitHub


liugddx commented on PR #25444:
URL: https://github.com/apache/doris/pull/25444#issuecomment-1773985417

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](CANCEL-ALTER-SYSTEM)decommission backend by ids [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #25441:
URL: https://github.com/apache/doris/pull/25441#issuecomment-1773989064

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.88 seconds
stream load tsv:  554 seconds loaded 74807831229 Bytes, about 128 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  33 seconds loaded 861443392 Bytes, about 24 
MB/s
insert into select:  29.0 seconds inserted 1000 Rows, about 
344K ops/s
storage size: 17162008239 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](alter backends)backend modify & drop & decommission by ids [doris]

2023-10-21 Thread via GitHub


doris-robot commented on PR #25444:
URL: https://github.com/apache/doris/pull/25444#issuecomment-1773990029

   (From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.8 seconds
stream load tsv:  555 seconds loaded 74807831229 Bytes, about 128 
MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc:  65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet:  32 seconds loaded 861443392 Bytes, about 25 
MB/s
insert into select:  29.0 seconds inserted 1000 Rows, about 
344K ops/s
storage size: 17162108587 Bytes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](code-style)add spotless plugin and remove checkstyle [doris]

2023-10-21 Thread via GitHub


liugddx commented on PR #25033:
URL: https://github.com/apache/doris/pull/25033#issuecomment-1773997988

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [feature](function) support ip function ipv4stringtonum(ordefault, ornull), inet_aton [doris]

2023-10-21 Thread via GitHub


github-actions[bot] commented on PR #25510:
URL: https://github.com/apache/doris/pull/25510#issuecomment-1774000468

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [Improvements](status) catch some error status [doris]

2023-10-21 Thread via GitHub


BiteThet commented on PR #25677:
URL: https://github.com/apache/doris/pull/25677#issuecomment-1774002400

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [enhancement](recover) support skipping missing version in select by session variable [doris]

2023-10-21 Thread via GitHub


xy720 commented on PR #25654:
URL: https://github.com/apache/doris/pull/25654#issuecomment-1774002887

   run p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



  1   2   >