Re: [I] [Bug] Doris Be node crash ,version:doris-1.2.7 [doris]
fsilent commented on issue #24228: URL: https://github.com/apache/doris/issues/24228#issuecomment-1773702069 hello, have you solved this problem? @liuchunhua -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [regresstion-test](jdbc catalog)Mariadb compatible test [doris]
zy-kkk commented on PR #25664: URL: https://github.com/apache/doris/pull/25664#issuecomment-1773719659 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [build] Update build-extension.yml [doris]
zy-kkk merged PR #25721: URL: https://github.com/apache/doris/pull/25721 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [build](extension) Update build-extension.yml (#25721)
This is an automated email from the ASF dual-hosted git repository. zykkk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 387a9c74489 [build](extension) Update build-extension.yml (#25721) 387a9c74489 is described below commit 387a9c74489c47930d3bb2506c70fdd06f0cefde Author: Jeffrey AuthorDate: Sat Oct 21 16:52:55 2023 +0800 [build](extension) Update build-extension.yml (#25721) --- .github/workflows/build-extension.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/build-extension.yml b/.github/workflows/build-extension.yml index 4a9fc5d38ef..d58e21a75dc 100644 --- a/.github/workflows/build-extension.yml +++ b/.github/workflows/build-extension.yml @@ -100,6 +100,7 @@ jobs: run: | git clone https://github.com/apache/doris-website.git website cd website + echo "[\"current\"]" > versions.json mkdir -p docs cp -R ../docs/en/docs/* docs/ cp -R ../docs/sidebars.json sidebars.json - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-2.0 updated: [fix](ut) fix fe ut for branch-2.0 (#25719)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.0 by this push: new 3389f3f9d1d [fix](ut) fix fe ut for branch-2.0 (#25719) 3389f3f9d1d is described below commit 3389f3f9d1d17f3018b0bc20970298a04a380324 Author: AKIRA <33112463+kikyou1...@users.noreply.github.com> AuthorDate: Sat Oct 21 18:31:36 2023 +0800 [fix](ut) fix fe ut for branch-2.0 (#25719) --- .../doris/statistics/AnalysisManagerTest.java | 62 ++ 1 file changed, 39 insertions(+), 23 deletions(-) diff --git a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisManagerTest.java b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisManagerTest.java index 636e32ea4e1..c995710da44 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisManagerTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisManagerTest.java @@ -25,6 +25,8 @@ import org.apache.doris.catalog.Column; import org.apache.doris.catalog.OlapTable; import org.apache.doris.catalog.PrimitiveType; import org.apache.doris.common.DdlException; +import org.apache.doris.statistics.AnalysisInfo.AnalysisType; +import org.apache.doris.statistics.AnalysisInfo.JobType; import org.apache.doris.statistics.AnalysisInfo.ScheduleType; import org.apache.doris.statistics.util.StatisticsUtil; @@ -45,10 +47,11 @@ import java.util.HashSet; import java.util.List; import java.util.Map; +// CHECKSTYLE OFF public class AnalysisManagerTest { @Test public void testUpdateTaskStatus(@Mocked BaseAnalysisTask task1, - @Mocked BaseAnalysisTask task2) { +@Mocked BaseAnalysisTask task2) { new MockUp() { @Mock @@ -61,12 +64,22 @@ public class AnalysisManagerTest { }; +new MockUp() { +@Mock +public String toString() { +return ""; +} +}; + AnalysisInfo job = new AnalysisInfoBuilder().setJobId(1) - .setState(AnalysisState.PENDING).setJobType(AnalysisInfo.JobType.MANUAL).build(); + .setState(AnalysisState.PENDING).setAnalysisType(AnalysisType.FUNDAMENTALS) +.setJobType(AnalysisInfo.JobType.MANUAL).build(); AnalysisInfo taskInfo1 = new AnalysisInfoBuilder().setJobId(1) -.setTaskId(2).setState(AnalysisState.PENDING).build(); + .setTaskId(2).setJobType(JobType.MANUAL).setAnalysisType(AnalysisType.FUNDAMENTALS) +.setState(AnalysisState.PENDING).build(); AnalysisInfo taskInfo2 = new AnalysisInfoBuilder().setJobId(1) -.setTaskId(3).setState(AnalysisState.PENDING).build(); + .setTaskId(3).setAnalysisType(AnalysisType.FUNDAMENTALS).setJobType(JobType.MANUAL) +.setState(AnalysisState.PENDING).build(); AnalysisManager manager = new AnalysisManager(); manager.replayCreateAnalysisJob(job); manager.replayCreateAnalysisTask(taskInfo1); @@ -139,15 +152,15 @@ public class AnalysisManagerTest { add("p2"); } }), new ArrayList() { -{ -add("c1"); -add("c2"); -} -}, new AnalyzeProperties(new HashMap() { -{ -put(AnalyzeProperties.PROPERTY_SYNC, "true"); -} -})); +{ +add("c1"); +add("c2"); +} +}, new AnalyzeProperties(new HashMap() { +{ +put(AnalyzeProperties.PROPERTY_SYNC, "true"); +} +})); AnalysisManager analysisManager = new AnalysisManager(); Assertions.assertNull(analysisManager.buildAndAssignJob(analyzeTblStmt)); @@ -228,16 +241,16 @@ public class AnalysisManagerTest { add("p2"); } }), new ArrayList() { -{ -add("c1"); -add("c2"); -} -}, new AnalyzeProperties(new HashMap() { -{ -put(AnalyzeProperties.PROPERTY_SYNC, "false"); -put(AnalyzeProperties.PROPERTY_PERIOD_SECONDS, "100"); -} -})); +{ +add("c1"); +add("c2"); +} +}, new AnalyzeProperties(new HashMap() { +{ +put(AnalyzeProperties.PROPERTY_SYNC, "false"); +put(AnalyzeProperties.PROPERTY_PERIOD_SECONDS, "100"); +} +})); AnalysisManager analysisManager = new AnalysisManager(
Re: [PR] [fix](stats) Fix ut [doris]
xiaokang merged PR #25719: URL: https://github.com/apache/doris/pull/25719 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](CANCEL-ALTER-SYSTEM)decommission backend by ids [doris]
liugddx commented on PR #25441: URL: https://github.com/apache/doris/pull/25441#issuecomment-1773748000 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](CANCEL-ALTER-SYSTEM)decommission backend by ids [doris]
doris-robot commented on PR #25441: URL: https://github.com/apache/doris/pull/25441#issuecomment-1773761297 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.39 seconds stream load tsv: 551 seconds loaded 74807831229 Bytes, about 129 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.0 seconds inserted 1000 Rows, about 344K ops/s storage size: 17162008636 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] Doris Be node crash ,version:doris-1.2.7 [doris]
liuchunhua commented on issue #24228: URL: https://github.com/apache/doris/issues/24228#issuecomment-1773761952 not yet -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](paimon)paimon catalog supports complex types [doris]
DongLiang-0 commented on PR #25364: URL: https://github.com/apache/doris/pull/25364#issuecomment-1773771412 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](paimon)paimon catalog supports complex types [doris]
doris-robot commented on PR #25364: URL: https://github.com/apache/doris/pull/25364#issuecomment-1773781759 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.81 seconds stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17162015400 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](streamload) refactor stream load executor [doris]
HHoflittlefish777 commented on PR #25615: URL: https://github.com/apache/doris/pull/25615#issuecomment-1773791004 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](streamload) refactor stream load executor [doris]
github-actions[bot] commented on PR #25615: URL: https://github.com/apache/doris/pull/25615#issuecomment-1773793191 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](streamload) refactor stream load executor [doris]
github-actions[bot] commented on PR #25615: URL: https://github.com/apache/doris/pull/25615#issuecomment-1773793855 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](streamload) refactor stream load executor [doris]
doris-robot commented on PR #25615: URL: https://github.com/apache/doris/pull/25615#issuecomment-1773798091 TeamCity be ut coverage result: Function Coverage: 37.05% (8303/22410) Line Coverage: 29.21% (66693/228326) Region Coverage: 27.84% (34626/124360) Branch Coverage: 24.43% (17586/71996) Coverage Report: http://coverage.selectdb-in.cc/coverage/aa21dc81218a83cf2addf9a11c1c0162676a1555_aa21dc81218a83cf2addf9a11c1c0162676a1555/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](streamload) refactor stream load executor [doris]
doris-robot commented on PR #25615: URL: https://github.com/apache/doris/pull/25615#issuecomment-1773803643 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.67 seconds stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17162439292 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](recover) support skipping missing version in select by session variable [doris]
github-actions[bot] commented on PR #25654: URL: https://github.com/apache/doris/pull/25654#issuecomment-1773809247 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [doc](pick)support microseconds/milliseconds func [doris]
xiaokang merged PR #25245: URL: https://github.com/apache/doris/pull/25245 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-2.0 updated: [doc](pick)support microseconds/milliseconds func (#25245)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.0 by this push: new a5d5a8bbe03 [doc](pick)support microseconds/milliseconds func (#25245) a5d5a8bbe03 is described below commit a5d5a8bbe0366b5f3eb1943929edf0233769062d Author: Liqf <109049295+lemonlit...@users.noreply.github.com> AuthorDate: Sat Oct 21 22:33:16 2023 +0800 [doc](pick)support microseconds/milliseconds func (#25245) --- docs/sidebars.json | 5 + 1 file changed, 5 insertions(+) diff --git a/docs/sidebars.json b/docs/sidebars.json index fb8982d199f..089f0022d75 100644 --- a/docs/sidebars.json +++ b/docs/sidebars.json @@ -360,6 +360,11 @@ "sql-manual/sql-functions/date-time-functions/date_format", "sql-manual/sql-functions/date-time-functions/datediff", "sql-manual/sql-functions/date-time-functions/microseconds_add", + "sql-manual/sql-functions/date-time-functions/microseconds-diff", + "sql-manual/sql-functions/date-time-functions/microseconds-sub", + "sql-manual/sql-functions/date-time-functions/milliseconds-add", + "sql-manual/sql-functions/date-time-functions/milliseconds-diff", + "sql-manual/sql-functions/date-time-functions/milliseconds-sub", "sql-manual/sql-functions/date-time-functions/minutes_add", "sql-manual/sql-functions/date-time-functions/minutes_diff", "sql-manual/sql-functions/date-time-functions/minutes_sub", - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](ColdHeatSeperation) Support to upload cold data to HDFS [doris]
morningman commented on PR #22048: URL: https://github.com/apache/doris/pull/22048#issuecomment-177386 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](log) Print correct queryId & instance & [doris]
xiaokang commented on PR #25532: URL: https://github.com/apache/doris/pull/25532#issuecomment-1773811397 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](stats) Enable auto sample by default [doris]
xiaokang commented on PR #25607: URL: https://github.com/apache/doris/pull/25607#issuecomment-1773811711 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) runtime filter generator meet npe when process set operation [doris]
xiaokang commented on PR #25612: URL: https://github.com/apache/doris/pull/25612#issuecomment-1773811843 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pick](Branch2.0) fix row count unconsistent when join ordering (#24589) [doris]
xiaokang commented on PR #25666: URL: https://github.com/apache/doris/pull/25666#issuecomment-1773812135 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pick](Branch2.0) generate left deep tree when stats is unknown [doris]
xiaokang commented on PR #25702: URL: https://github.com/apache/doris/pull/25702#issuecomment-1773812298 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)pick part of "[opt](nereids) enable runtime filter prune by default #24717" [doris]
xiaokang commented on PR #25642: URL: https://github.com/apache/doris/pull/25642#issuecomment-1773811996 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](profile) set datalake profile level as 1 [doris]
github-actions[bot] commented on PR #25686: URL: https://github.com/apache/doris/pull/25686#issuecomment-1773812457 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](sync version) fe sync version with be [doris]
xiaokang commented on PR #25718: URL: https://github.com/apache/doris/pull/25718#issuecomment-1773812469 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](profile) set datalake profile level as 1 [doris]
github-actions[bot] commented on PR #25686: URL: https://github.com/apache/doris/pull/25686#issuecomment-1773812477 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](ColdHeatSeperation) Support to upload cold data to HDFS [doris]
github-actions[bot] commented on PR #22048: URL: https://github.com/apache/doris/pull/22048#issuecomment-1773813573 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](log) Print correct queryId & instance & [doris]
github-actions[bot] commented on PR #25532: URL: https://github.com/apache/doris/pull/25532#issuecomment-1773814005 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](sync version) fe sync version with be [doris]
github-actions[bot] commented on PR #25718: URL: https://github.com/apache/doris/pull/25718#issuecomment-1773814785 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Outfile) Export `DateTimev2` type of doris to ORC's `TimeStamp` type [doris]
morningman commented on code in PR #25470: URL: https://github.com/apache/doris/pull/25470#discussion_r1367740540 ## be/src/vec/runtime/vorc_transformer.cpp: ## @@ -94,7 +94,10 @@ VOrcTransformer::VOrcTransformer(doris::io::FileWriter* file_writer, : VFileFormatTransformer(output_vexpr_ctxs, output_object_data), _file_writer(file_writer), _write_options(new orc::WriterOptions()), - _schema_str(schema) {} + _schema_str(schema) { +_write_options->setTimezoneName("Asia/Shanghai"); Review Comment: Should it be a fixed value? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](filecache) do not sync segment data into storage system [doris]
morningman commented on PR #25691: URL: https://github.com/apache/doris/pull/25691#issuecomment-1773817528 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](filecache) do not sync segment data into storage system [doris]
github-actions[bot] commented on PR #25691: URL: https://github.com/apache/doris/pull/25691#issuecomment-1773818189 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](filecache) do not sync segment data into storage system [doris]
github-actions[bot] commented on PR #25691: URL: https://github.com/apache/doris/pull/25691#issuecomment-1773818201 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](ColdHeatSeperation) Support to upload cold data to HDFS [doris]
doris-robot commented on PR #22048: URL: https://github.com/apache/doris/pull/22048#issuecomment-1773821377 TeamCity be ut coverage result: Function Coverage: 37.05% (8303/22410) Line Coverage: 29.20% (66686/228341) Region Coverage: 27.84% (34624/124371) Branch Coverage: 24.42% (17586/72002) Coverage Report: http://coverage.selectdb-in.cc/coverage/91d8ff73cc2d59f29861a1c84e14936093a88471_91d8ff73cc2d59f29861a1c84e14936093a88471/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](auth) support show all property [doris]
morningman commented on code in PR #25645: URL: https://github.com/apache/doris/pull/25645#discussion_r1367742106 ## docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-PROPERTY.md: ## @@ -62,6 +66,26 @@ mysql> show property like'%connection%'; Attribute value. + +```sql +mysql> show all property like "%connection%"; Review Comment: better use `show all properties` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](auth)Fix no priv to insert to mysql catalog [doris]
github-actions[bot] commented on PR #25678: URL: https://github.com/apache/doris/pull/25678#issuecomment-1773823926 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](auth)Fix no priv to insert to mysql catalog [doris]
github-actions[bot] commented on PR #25678: URL: https://github.com/apache/doris/pull/25678#issuecomment-1773823939 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](scanner_schedule) reduce memory consumption of scanner #24199 [doris]
xiaokang commented on PR #25547: URL: https://github.com/apache/doris/pull/25547#issuecomment-1773825452 convert it to draft since an bug is reported related to this pr on master. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](dynamic) Fix error reporting when dynamic partition properties contain incorrect attributes [doris]
Zhiyu-h commented on PR #25373: URL: https://github.com/apache/doris/pull/25373#issuecomment-1773825590 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)(create-table) fix bug that replication num is not set when create table with no property [doris]
morningman merged PR #25651: URL: https://github.com/apache/doris/pull/25651 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch master updated: [fix](nereids)(create-table) fix bug that replication num is not set when create table with no property (#25651)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 13780e4827d [fix](nereids)(create-table) fix bug that replication num is not set when create table with no property (#25651) 13780e4827d is described below commit 13780e4827db38494f2f3fb168f3195163a18fc1 Author: Mingyu Chen AuthorDate: Sat Oct 21 23:15:08 2023 +0800 [fix](nereids)(create-table) fix bug that replication num is not set when create table with no property (#25651) When executing create partitioned table with Nereids, and replication_num property is not set, the replication number will be 0, so the tablet will has no replica. --- .../org/apache/doris/analysis/CreateTableStmt.java | 75 +- .../apache/doris/common/util/PropertyAnalyzer.java | 72 ++ .../doris/nereids/parser/LogicalPlanBuilder.java | 11 ++- .../trees/plans/commands/info/CreateTableInfo.java | 45 ++--- .../plans/commands/info/FixedRangePartition.java | 10 +- .../trees/plans/commands/info/InPartition.java | 9 +- .../plans/commands/info/LessThanPartition.java | 10 +- .../plans/commands/info/PartitionDefinition.java | 22 - .../trees/plans/commands/info/StepPartition.java | 9 +- .../org/apache/doris/catalog/CreateTableTest.java | 110 + .../org/apache/doris/common/ExceptionChecker.java | 1 + .../apache/doris/utframe/TestWithFeService.java| 37 +-- 12 files changed, 240 insertions(+), 171 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java index a03884f5c6e..f56de294d98 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java +++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateTableStmt.java @@ -20,13 +20,11 @@ package org.apache.doris.analysis; import org.apache.doris.analysis.IndexDef.IndexType; import org.apache.doris.catalog.AggregateType; import org.apache.doris.catalog.Column; -import org.apache.doris.catalog.DatabaseIf; import org.apache.doris.catalog.DistributionInfo; import org.apache.doris.catalog.Env; import org.apache.doris.catalog.Index; import org.apache.doris.catalog.KeysType; import org.apache.doris.catalog.PrimitiveType; -import org.apache.doris.catalog.ReplicaAllocation; import org.apache.doris.catalog.Type; import org.apache.doris.common.AnalysisException; import org.apache.doris.common.Config; @@ -41,7 +39,6 @@ import org.apache.doris.common.util.ParseUtil; import org.apache.doris.common.util.PrintableMap; import org.apache.doris.common.util.PropertyAnalyzer; import org.apache.doris.common.util.Util; -import org.apache.doris.datasource.CatalogIf; import org.apache.doris.external.elasticsearch.EsUtil; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.qe.ConnectContext; @@ -51,7 +48,6 @@ import com.google.common.collect.Lists; import com.google.common.collect.Maps; import com.google.common.collect.Sets; import org.apache.commons.collections.CollectionUtils; -import org.apache.commons.lang3.StringUtils; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -559,7 +555,8 @@ public class CreateTableStmt extends DdlStmt { if (engineName.equals("olap")) { // before analyzing partition, handle the replication allocation info -properties = rewriteReplicaAllocationProperties(properties); +properties = PropertyAnalyzer.rewriteReplicaAllocationProperties( +tableName.getCtl(), tableName.getDb(), properties); // analyze partition if (partitionDesc != null) { if (partitionDesc instanceof ListPartitionDesc || partitionDesc instanceof RangePartitionDesc @@ -650,74 +647,6 @@ public class CreateTableStmt extends DdlStmt { } } -private Map rewriteReplicaAllocationProperties(Map properties) -throws AnalysisException { -if (Config.force_olap_table_replication_num <= 0) { -return rewriteReplicaAllocationPropertiesByDatabase(properties); -} -// if force_olap_table_replication_num is set, use this value to rewrite the replication_num or -// replication_allocation properties -Map newProperties = properties; -if (newProperties == null) { -newProperties = Maps.newHashMap(); -} -boolean rewrite = false; -if (newProperties.containsKey(PropertyAnalyzer.PROPERTIES_REPLICATION_NUM)) { -newProperties.put(PropertyAnalyzer.PROPERTIES_REPLICATION_NUM, -String.valueOf(Config.force_olap_table_replication_num)); -rewrite = true; -} -if (newProp
Re: [PR] [feat](stats) Enable auto sample by default [doris]
hello-stephen commented on PR #25607: URL: https://github.com/apache/doris/pull/25607#issuecomment-1773827449 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.26 seconds stream load tsv: 573 seconds loaded 74807831229 Bytes, about 124 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.5 seconds inserted 1000 Rows, about 338K ops/s storage size: 17163856179 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pick](Branch2.0) fix row count unconsistent when join ordering (#24589) [doris]
hello-stephen commented on PR #25666: URL: https://github.com/apache/doris/pull/25666#issuecomment-1773827506 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.06 seconds stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 29.9 seconds inserted 1000 Rows, about 334K ops/s storage size: 17167870587 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) runtime filter generator meet npe when process set operation [doris]
hello-stephen commented on PR #25612: URL: https://github.com/apache/doris/pull/25612#issuecomment-1773827571 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.96 seconds stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.5 seconds inserted 1000 Rows, about 338K ops/s storage size: 17168027630 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](log) Print correct queryId & instance & [doris]
hello-stephen commented on PR #25532: URL: https://github.com/apache/doris/pull/25532#issuecomment-1773827747 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 48.74 seconds stream load tsv: 574 seconds loaded 74807831229 Bytes, about 124 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.5 seconds inserted 1000 Rows, about 338K ops/s storage size: 17163160328 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](ColdHeatSeperation) Support to upload cold data to HDFS [doris]
doris-robot commented on PR #22048: URL: https://github.com/apache/doris/pull/22048#issuecomment-1773827823 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.85 seconds stream load tsv: 553 seconds loaded 74807831229 Bytes, about 129 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.7 seconds inserted 1000 Rows, about 348K ops/s storage size: 17162151206 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pick](Branch2.0) generate left deep tree when stats is unknown [doris]
hello-stephen commented on PR #25702: URL: https://github.com/apache/doris/pull/25702#issuecomment-1773831287 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.95 seconds stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s insert into select: 29.2 seconds inserted 1000 Rows, about 342K ops/s storage size: 17162441392 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](filecache) do not sync segment data into storage system [doris]
doris-robot commented on PR #25691: URL: https://github.com/apache/doris/pull/25691#issuecomment-1773831836 TeamCity be ut coverage result: Function Coverage: 37.04% (8301/22410) Line Coverage: 29.20% (66676/228328) Region Coverage: 27.84% (34622/124363) Branch Coverage: 24.43% (17587/71998) Coverage Report: http://coverage.selectdb-in.cc/coverage/62048a06842b474bf86fe51c6e1a974bbc0c658b_62048a06842b474bf86fe51c6e1a974bbc0c658b/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](exec) Fix date dict infinite loop. [doris]
kaka11chen commented on PR #25570: URL: https://github.com/apache/doris/pull/25570#issuecomment-1773837048 run buildalll -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](alter backends)backend modify & drop & decommission by ids [doris]
liugddx commented on PR #25444: URL: https://github.com/apache/doris/pull/25444#issuecomment-1773838359 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](exec) Fix date dict infinite loop. [doris]
github-actions[bot] commented on code in PR #25570: URL: https://github.com/apache/doris/pull/25570#discussion_r1367751431 ## be/test/vec/runtime/vdatetime_value_test.cpp: ## @@ -570,4 +570,172 @@ TEST(VDateTimeValueTest, date_v2_to_string_test) { } } +TEST(VDateTimeValueTest, date_v2_daynr_test) { +{ +DateV2Value date_v2; +// 1970/01/01 +EXPECT_TRUE(date_v2.get_date_from_daynr(719528)); +EXPECT_TRUE(date_v2.year() == 1970); +EXPECT_TRUE(date_v2.month() == 1); +EXPECT_TRUE(date_v2.day() == 1); +EXPECT_TRUE(date_v2.hour() == 0); +EXPECT_TRUE(date_v2.minute() == 0); +EXPECT_TRUE(date_v2.second() == 0); +EXPECT_TRUE(date_v2.microsecond() == 0); +EXPECT_TRUE(doris::calc_daynr(1970, 1, 1) == 719528); +EXPECT_TRUE(date_day_offset_dict::get().get_dict_init()); +EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_calc_daynr(1970)); + EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_daynr_to_date(719528)); +} + +{ +DateV2Value date_v2; +// 1969/12/31 +EXPECT_TRUE(date_v2.get_date_from_daynr(719527)); +EXPECT_TRUE(date_v2.year() == 1969); +EXPECT_TRUE(date_v2.month() == 12); +EXPECT_TRUE(date_v2.day() == 31); +EXPECT_TRUE(date_v2.hour() == 0); +EXPECT_TRUE(date_v2.minute() == 0); +EXPECT_TRUE(date_v2.second() == 0); +EXPECT_TRUE(date_v2.microsecond() == 0); +EXPECT_TRUE(doris::calc_daynr(1969, 12, 31) == 719527); +EXPECT_TRUE(date_day_offset_dict::get().get_dict_init()); +EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_calc_daynr(1969)); + EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_daynr_to_date(719527)); +} + +{ +DateV2Value date_v2; +// 1900/01/01 +EXPECT_TRUE(date_v2.get_date_from_daynr(693961)); +EXPECT_TRUE(date_v2.year() == 1900); +EXPECT_TRUE(date_v2.month() == 1); +EXPECT_TRUE(date_v2.day() == 1); +EXPECT_TRUE(date_v2.hour() == 0); +EXPECT_TRUE(date_v2.minute() == 0); +EXPECT_TRUE(date_v2.second() == 0); +EXPECT_TRUE(date_v2.microsecond() == 0); +EXPECT_TRUE(doris::calc_daynr(1900, 1, 1) == 693961); +EXPECT_TRUE(date_day_offset_dict::get().get_dict_init()); +EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_calc_daynr(1900)); + EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_daynr_to_date(693961)); +} + +{ +DateV2Value date_v2; +// 1899/12/31 +EXPECT_TRUE(date_v2.get_date_from_daynr(693960)); +EXPECT_TRUE(date_v2.year() == 1899); +EXPECT_TRUE(date_v2.month() == 12); +EXPECT_TRUE(date_v2.day() == 31); +EXPECT_TRUE(date_v2.hour() == 0); +EXPECT_TRUE(date_v2.minute() == 0); +EXPECT_TRUE(date_v2.second() == 0); +EXPECT_TRUE(date_v2.microsecond() == 0); +EXPECT_TRUE(doris::calc_daynr(1899, 12, 31) == 693960); +EXPECT_TRUE(date_day_offset_dict::get().get_dict_init()); + EXPECT_FALSE(date_day_offset_dict::get().can_speed_up_calc_daynr(1899)); + EXPECT_FALSE(date_day_offset_dict::get().can_speed_up_daynr_to_date(693960)); +} + +{ +DateV2Value date_v2; +// 2039/12/31 +EXPECT_TRUE(date_v2.get_date_from_daynr(745094)); +EXPECT_TRUE(date_v2.year() == 2039); +EXPECT_TRUE(date_v2.month() == 12); +EXPECT_TRUE(date_v2.day() == 31); +EXPECT_TRUE(date_v2.hour() == 0); +EXPECT_TRUE(date_v2.minute() == 0); +EXPECT_TRUE(date_v2.second() == 0); +EXPECT_TRUE(date_v2.microsecond() == 0); +EXPECT_TRUE(doris::calc_daynr(2039, 12, 31) == 745094); +EXPECT_TRUE(date_day_offset_dict::get().get_dict_init()); +EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_calc_daynr(2039)); + EXPECT_TRUE(date_day_offset_dict::get().can_speed_up_daynr_to_date(745094)); +} + +{ +DateV2Value date_v2; +// 2040/01/01 +EXPECT_TRUE(date_v2.get_date_from_daynr(745095)); +EXPECT_TRUE(date_v2.year() == 2040); +EXPECT_TRUE(date_v2.month() == 1); +EXPECT_TRUE(date_v2.day() == 1); +EXPECT_TRUE(date_v2.hour() == 0); +EXPECT_TRUE(date_v2.minute() == 0); +EXPECT_TRUE(date_v2.second() == 0); +EXPECT_TRUE(date_v2.microsecond() == 0); +EXPECT_TRUE(doris::calc_daynr(2040, 01, 01) == 745095); +EXPECT_TRUE(date_day_offset_dict::get().get_dict_init()); + EXPECT_FALSE(date_day_offset_dict::get().can_speed_up_calc_daynr(2040)); + EXPECT_FALSE(date_day_offset_dict::get().can_speed_up_daynr_to_date(745095)); +} + +{ +DateV2Value date_v2; +// /01/01 +EXPECT_TRUE(date_v2.get_date_from_daynr(1)); +EXPECT_TRUE(date_v2.year() == 0); +EXPECT_TRUE(date_v2.month() == 1); +
Re: [PR] [chore](log) Print correct queryId & instance & [doris]
xiaokang merged PR #25532: URL: https://github.com/apache/doris/pull/25532 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-2.0 updated: [chore](log) Print correct queryId & instance (#25532)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.0 by this push: new 6f9e39e4093 [chore](log) Print correct queryId & instance (#25532) 6f9e39e4093 is described below commit 6f9e39e40930a90d67e2574fa89832f905dbc210 Author: zhiqiang AuthorDate: Sat Oct 21 11:42:59 2023 -0500 [chore](log) Print correct queryId & instance (#25532) --- be/src/pipeline/pipeline_fragment_context.cpp | 4 ++-- be/src/runtime/fragment_mgr.cpp | 4 ++-- be/src/runtime/plan_fragment_executor.cpp | 10 +- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/be/src/pipeline/pipeline_fragment_context.cpp b/be/src/pipeline/pipeline_fragment_context.cpp index f18f437ba47..cd529ee835d 100644 --- a/be/src/pipeline/pipeline_fragment_context.cpp +++ b/be/src/pipeline/pipeline_fragment_context.cpp @@ -211,8 +211,8 @@ Status PipelineFragmentContext::prepare(const doris::TPipelineFragmentParams& re } LOG_INFO("PipelineFragmentContext::prepare") -.tag("query_id", _query_id) -.tag("instance_id", local_params.fragment_instance_id) +.tag("query_id", print_id(_query_id)) +.tag("instance_id", print_id(local_params.fragment_instance_id)) .tag("backend_num", local_params.backend_num) .tag("pthread_id", (uintptr_t)pthread_self()); diff --git a/be/src/runtime/fragment_mgr.cpp b/be/src/runtime/fragment_mgr.cpp index 409a32a128d..9f38e91ff15 100644 --- a/be/src/runtime/fragment_mgr.cpp +++ b/be/src/runtime/fragment_mgr.cpp @@ -526,8 +526,8 @@ void FragmentMgr::_exec_actual(std::shared_ptr exec_state, #endif LOG_INFO(func_name) -.tag("query_id", exec_state->query_id()) -.tag("instance_id", exec_state->fragment_instance_id()) +.tag("query_id", print_id(exec_state->query_id())) +.tag("instance_id", print_id(exec_state->fragment_instance_id())) .tag("pthread_id", (uintptr_t)pthread_self()); Status st = exec_state->execute(); diff --git a/be/src/runtime/plan_fragment_executor.cpp b/be/src/runtime/plan_fragment_executor.cpp index 77d15938303..de6f5f55810 100644 --- a/be/src/runtime/plan_fragment_executor.cpp +++ b/be/src/runtime/plan_fragment_executor.cpp @@ -113,8 +113,8 @@ Status PlanFragmentExecutor::prepare(const TExecPlanFragmentParams& request, _query_id = params.query_id; LOG_INFO("PlanFragmentExecutor::prepare") -.tag("query_id", _query_id) -.tag("instance_id", params.fragment_instance_id) +.tag("query_id", print_id(_query_id)) +.tag("instance_id", print_id(params.fragment_instance_id)) .tag("backend_num", request.backend_num) .tag("pthread_id", (uintptr_t)pthread_self()); // VLOG_CRITICAL << "request:\n" << apache::thrift::ThriftDebugString(request); @@ -249,8 +249,8 @@ Status PlanFragmentExecutor::prepare(const TExecPlanFragmentParams& request, Status PlanFragmentExecutor::open() { int64_t mem_limit = _runtime_state->query_mem_tracker()->limit(); LOG_INFO("PlanFragmentExecutor::open") -.tag("query_id", _query_id) -.tag("instance_id", _runtime_state->fragment_instance_id()) +.tag("query_id", print_id(_query_id)) +.tag("instance_id", print_id(_runtime_state->fragment_instance_id())) .tag("mem_limit", PrettyPrinter::print(mem_limit, TUnit::BYTES)); // we need to start the profile-reporting thread before calling Open(), since it @@ -488,7 +488,7 @@ void PlanFragmentExecutor::stop_report_thread() { void PlanFragmentExecutor::cancel(const PPlanFragmentCancelReason& reason, const std::string& msg) { LOG_INFO("PlanFragmentExecutor::cancel") -.tag("query_id", _query_id) +.tag("query_id", print_id(_query_id)) .tag("instance_id", _runtime_state->fragment_instance_id()) .tag("reason", reason) .tag("error message", msg); - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) runtime filter generator meet npe when process set operation [doris]
xiaokang merged PR #25612: URL: https://github.com/apache/doris/pull/25612 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-2.0 updated: [fix](nereids) runtime filter generator meet npe when process set operation (#25612)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.0 by this push: new 5a21a03b279 [fix](nereids) runtime filter generator meet npe when process set operation (#25612) 5a21a03b279 is described below commit 5a21a03b279a5e7bd85ef2c5a03028fbec3824d4 Author: minghong AuthorDate: Sun Oct 22 00:44:18 2023 +0800 [fix](nereids) runtime filter generator meet npe when process set operation (#25612) --- .../apache/doris/nereids/processor/post/RuntimeFilterGenerator.java | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java index 6fd2f8838c2..0243326b106 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java @@ -368,8 +368,10 @@ public class RuntimeFilterGenerator extends PlanPostProcessor { break; } } -Preconditions.checkState(projIndex >= 0 -&& projIndex < project.getProjects().size()); +if (projIndex < 0 +|| projIndex >= project.getProjects().size()) { +continue; +} NamedExpression targetExpr = (NamedExpression) project.getProjects().get(projIndex); - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)pick part of "[opt](nereids) enable runtime filter prune by default #24717" [doris]
xiaokang commented on PR #25642: URL: https://github.com/apache/doris/pull/25642#issuecomment-1773855755 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pick](Branch2.0) fix row count unconsistent when join ordering (#24589) [doris]
xiaokang merged PR #25666: URL: https://github.com/apache/doris/pull/25666 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-2.0 updated: [fix](Nereids) fix row count unconsistent when join ordering (#24589) (#25666)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.0 by this push: new f2f8a6afcae [fix](Nereids) fix row count unconsistent when join ordering (#24589) (#25666) f2f8a6afcae is described below commit f2f8a6afcae582fc0931efe9e5f61f271043b896 Author: 谢健 AuthorDate: Sun Oct 22 00:45:52 2023 +0800 [fix](Nereids) fix row count unconsistent when join ordering (#24589) (#25666) In the context of reorder join, when a new plan is generated, it may include a project operation. In this case, the newly generated join root and the original join root will no longer be in the same group. To avoid inconsistencies in the statistics between these two groups, we keep the child group's row count unchanged when the parent group expression is a project operation. --- .../nereids/jobs/cascades/DeriveStatsJob.java | 12 +++ .../nereids_tpcds_shape_sf100_p0/shape/query18.out | 14 ++-- .../nereids_tpcds_shape_sf100_p0/shape/query59.out | 35 + .../nereids_tpcds_shape_sf100_p0/shape/query61.out | 36 + .../nereids_tpcds_shape_sf100_p0/shape/query71.out | 73 +- .../nereids_tpcds_shape_sf100_p0/shape/query72.out | 48 +++- .../nereids_tpcds_shape_sf100_p0/shape/query99.out | 8 +- .../data/nereids_tpch_shape_sf1000_p0/shape/q8.out | 49 ++-- .../nereids_tpcds_shape_sf100_p0/rf/ds_rf18.groovy | 2 +- .../nereids_tpcds_shape_sf100_p0/rf/ds_rf59.groovy | 83 +++-- .../nereids_tpcds_shape_sf100_p0/rf/ds_rf61.groovy | 86 +++--- .../nereids_tpcds_shape_sf100_p0/rf/ds_rf72.groovy | 2 +- .../nereids_tpcds_shape_sf100_p0/rf/ds_rf99.groovy | 2 +- .../nereids_tpch_shape_sf1000_p0/rf/h_rf8.groovy | 2 +- 14 files changed, 261 insertions(+), 191 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/DeriveStatsJob.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/DeriveStatsJob.java index cfe952c0f26..ec65b9af14d 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/DeriveStatsJob.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/cascades/DeriveStatsJob.java @@ -28,6 +28,7 @@ import org.apache.doris.nereids.metrics.consumer.LogConsumer; import org.apache.doris.nereids.metrics.event.StatsStateEvent; import org.apache.doris.nereids.stats.StatsCalculator; import org.apache.doris.nereids.trees.expressions.CTEId; +import org.apache.doris.nereids.trees.plans.algebra.Project; import org.apache.doris.qe.ConnectContext; import org.apache.doris.statistics.Statistics; @@ -116,6 +117,17 @@ public class DeriveStatsJob extends Job { context.getCascadesContext().getConnectContext().getTotalHistogramMap() .putAll(statsCalculator.getTotalHistogramMap()); } + +if (groupExpression.getPlan() instanceof Project) { +// In the context of reorder join, when a new plan is generated, it may include a project operation. +// In this case, the newly generated join root and the original join root will no longer be in the +// same group. To avoid inconsistencies in the statistics between these two groups, we keep the +// child group's row count unchanged when the parent group expression is a project operation. +double parentRowCount = groupExpression.getOwnerGroup().getStatistics().getRowCount(); +groupExpression.children().forEach(g -> g.setStatistics( + g.getStatistics().withRowCountAndEnforceValid(parentRowCount)) +); +} } } } diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query18.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query18.out index f7e24486fd6..2ce64b0b73c 100644 --- a/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query18.out +++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/shape/query18.out @@ -10,15 +10,15 @@ PhysicalResultSink --hashAgg[LOCAL] PhysicalRepeat --PhysicalProject -hashJoin[INNER_JOIN](customer.c_current_cdemo_sk = cd2.cd_demo_sk) +hashJoin[INNER_JOIN](catalog_sales.cs_item_sk = item.i_item_sk) +--PhysicalProject +PhysicalOlapScan[item] --PhysicalDistribute PhysicalProject ---PhysicalOlapScan[customer_demographics] ---PhysicalDistribute -PhysicalProject ---hashJoin[INNER_JOIN](catalog_sales.cs_item_sk = item.i_item_sk) -PhysicalProject
Re: [PR] [pick](Branch2.0) generate left deep tree when stats is unknown [doris]
xiaokang merged PR #25702: URL: https://github.com/apache/doris/pull/25702 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-2.0 updated: [pick](Branch2.0) generate left deep tree when stats is unknown (#25702)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.0 by this push: new ff786d05fb4 [pick](Branch2.0) generate left deep tree when stats is unknown (#25702) ff786d05fb4 is described below commit ff786d05fb4d05aee444a267883fb7998f8ea91f Author: 谢健 AuthorDate: Sun Oct 22 00:46:43 2023 +0800 [pick](Branch2.0) generate left deep tree when stats is unknown (#25702) --- .../org/apache/doris/nereids/cost/CostModelV1.java | 43 +- .../apache/doris/nereids/stats/JoinEstimation.java | 17 - .../trees/plans/physical/PhysicalHashJoin.java | 8 .../plans/physical/PhysicalNestedLoopJoin.java | 8 4 files changed, 74 insertions(+), 2 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java index aa8f4d6cc7c..2aca6017b7b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/cost/CostModelV1.java @@ -22,6 +22,7 @@ import org.apache.doris.nereids.properties.DistributionSpec; import org.apache.doris.nereids.properties.DistributionSpecGather; import org.apache.doris.nereids.properties.DistributionSpecHash; import org.apache.doris.nereids.properties.DistributionSpecReplicated; +import org.apache.doris.nereids.trees.expressions.Slot; import org.apache.doris.nereids.trees.plans.Plan; import org.apache.doris.nereids.trees.plans.physical.PhysicalAssertNumRows; import org.apache.doris.nereids.trees.plans.physical.PhysicalDeferMaterializeOlapScan; @@ -311,17 +312,53 @@ class CostModelV1 extends PlanVisitor { } // TODO: since the outputs rows may expand a lot, penalty on it will cause bc never be chosen. // will refine this in next generation cost model. +if (isStatsUnknown(physicalHashJoin, buildStats, probeStats)) { +// forbid broadcast join when stats is unknown +return CostV1.of(rightRowCount * buildSideFactor + 1 / leftRowCount, +rightRowCount, +0 +); +} return CostV1.of(leftRowCount + rightRowCount * buildSideFactor + outputRowCount * probeSideFactor, rightRowCount, 0 ); } +if (isStatsUnknown(physicalHashJoin, buildStats, probeStats)) { +return CostV1.of(rightRowCount + 1 / leftRowCount, +rightRowCount, +0); +} return CostV1.of(leftRowCount + rightRowCount + outputRowCount, rightRowCount, 0 ); } +private boolean isStatsUnknown(PhysicalHashJoin join, +Statistics build, Statistics probe) { +for (Slot slot : join.getConditionSlot()) { +if ((build.columnStatistics().containsKey(slot) && !build.columnStatistics().get(slot).isUnKnown) +|| (probe.columnStatistics().containsKey(slot) && !probe.columnStatistics().get(slot).isUnKnown)) { +continue; +} +return true; +} +return false; +} + +private boolean isStatsUnknown(PhysicalNestedLoopJoin join, +Statistics build, Statistics probe) { +for (Slot slot : join.getConditionSlot()) { +if ((build.columnStatistics().containsKey(slot) && !build.columnStatistics().get(slot).isUnKnown) +|| (probe.columnStatistics().containsKey(slot) && !probe.columnStatistics().get(slot).isUnKnown)) { +continue; +} +return true; +} +return false; +} + @Override public Cost visitPhysicalNestedLoopJoin( PhysicalNestedLoopJoin nestedLoopJoin, @@ -330,7 +367,11 @@ class CostModelV1 extends PlanVisitor { Preconditions.checkState(context.arity() == 2); Statistics leftStatistics = context.getChildStatistics(0); Statistics rightStatistics = context.getChildStatistics(1); - +if (isStatsUnknown(nestedLoopJoin, leftStatistics, rightStatistics)) { +return CostV1.of(rightStatistics.getRowCount() + 1 / leftStatistics.getRowCount(), +rightStatistics.getRowCount(), +0); +} return CostV1.of( leftStatistics.getRowCount() * rightStatistics.getRowCount(), rightStatistics.getRowCount(), diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java index ef4575e3308..0498d68d793 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/
Re: [PR] [improvement](sync version) fe sync version with be [doris]
xiaokang merged PR #25718: URL: https://github.com/apache/doris/pull/25718 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-2.0 updated: [improvement](sync version) fe sync version with be (#25718)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.0 by this push: new 6fe9d5d1af9 [improvement](sync version) fe sync version with be (#25718) 6fe9d5d1af9 is described below commit 6fe9d5d1af9b9d5475a04925aea5f9e365108255 Author: yujun AuthorDate: Sun Oct 22 00:47:56 2023 +0800 [improvement](sync version) fe sync version with be (#25718) --- be/src/http/action/debug_point_action.cpp | 19 ++- be/src/util/debug_points.cpp | 39 +++-- be/src/util/debug_points.h | 72 - be/test/util/debug_points_test.cpp | 40 + .../java/org/apache/doris/catalog/Replica.java | 36 - .../apache/doris/catalog/TabletInvertedIndex.java | 22 ++- .../org/apache/doris/clone/TabletSchedCtx.java | 7 + .../apache/doris/common/util/DebugPointUtil.java | 95 +-- .../apache/doris/httpv2/rest/DebugPointAction.java | 25 ++- .../java/org/apache/doris/master/MasterImpl.java | 9 +- .../org/apache/doris/master/ReportHandler.java | 42 +++-- .../org/apache/doris/clone/RepairVersionTest.java | 177 + .../doris/common/util/DebugPointUtilTest.java | 18 +++ .../apache/doris/utframe/TestWithFeService.java| 2 +- 14 files changed, 533 insertions(+), 70 deletions(-) diff --git a/be/src/http/action/debug_point_action.cpp b/be/src/http/action/debug_point_action.cpp index 08b1e116b2b..04aa38efaa4 100644 --- a/be/src/http/action/debug_point_action.cpp +++ b/be/src/http/action/debug_point_action.cpp @@ -21,6 +21,7 @@ #include "http/http_channel.h" #include "http/http_status.h" #include "util/debug_points.h" +#include "util/time.h" namespace doris { @@ -43,17 +44,16 @@ void BaseDebugPointAction::handle(HttpRequest* req) { } Status AddDebugPointAction::_handle(HttpRequest* req) { -std::string debug_point = req->param("debug_point"); +std::string name = req->param("debug_point"); std::string execute = req->param("execute"); std::string timeout = req->param("timeout"); -if (debug_point.empty()) { +if (name.empty()) { return Status::InternalError("Empty debug point name"); } -int64_t execute_limit = -1; -int64_t timeout_second = -1; +auto debug_point = std::make_shared(); try { if (!execute.empty()) { -execute_limit = std::stol(execute); +debug_point->execute_limit = std::stol(execute); } } catch (const std::exception& e) { return Status::InternalError("Invalid execute limit format, execute {}, err {}", execute, @@ -61,14 +61,19 @@ Status AddDebugPointAction::_handle(HttpRequest* req) { } try { if (!timeout.empty()) { -timeout_second = std::stol(timeout); +int64_t timeout_second = std::stol(timeout); +if (timeout_second > 0) { +debug_point->expire_ms = MonotonicMillis() + timeout_second * MILLIS_PER_SEC; +} } } catch (const std::exception& e) { return Status::InternalError("Invalid timeout format, timeout {}, err {}", timeout, e.what()); } -DebugPoints::instance()->add(debug_point, execute_limit, timeout_second); +debug_point->params = *(req->params()); + +DebugPoints::instance()->add(name, debug_point); return Status::OK(); } diff --git a/be/src/util/debug_points.cpp b/be/src/util/debug_points.cpp index 587f8c944a3..43bb39df9a4 100644 --- a/be/src/util/debug_points.cpp +++ b/be/src/util/debug_points.cpp @@ -30,37 +30,42 @@ DebugPoints* DebugPoints::instance() { } bool DebugPoints::is_enable(const std::string& name) { +return get_debug_point(name) != nullptr; +} + +std::shared_ptr DebugPoints::get_debug_point(const std::string& name) { if (!config::enable_debug_points) { -return false; +return nullptr; } auto map_ptr = std::atomic_load_explicit(&_debug_points, std::memory_order_relaxed); auto it = map_ptr->find(name); if (it == map_ptr->end()) { -return false; +return nullptr; } -auto& debug_point = *(it->second); -if ((debug_point.expire_ms > 0 && MonotonicMillis() >= debug_point.expire_ms) || -(debug_point.execute_limit > 0 && - debug_point.execute_num.fetch_add(1, std::memory_order_relaxed) >= - debug_point.execute_limit)) { +auto debug_point = it->second; +if ((debug_point->expire_ms > 0 && MonotonicMillis() >= debug_point->expire_ms) || +(debug_point->execute_limit > 0 && + debug_point->execute_num.fetch_add(1, std::memory_order_relaxed) >= + debug_point->execute_limit)) { remove(name); -return false; +return nullptr; } -
[doris] branch branch-2.0 updated: [feat](stats) Enable auto sample by default (#25607)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.0 by this push: new 41fe511ae8d [feat](stats) Enable auto sample by default (#25607) 41fe511ae8d is described below commit 41fe511ae8db559b03baf42c6440c21edbfd8e41 Author: AKIRA <33112463+kikyou1...@users.noreply.github.com> AuthorDate: Sun Oct 22 00:48:49 2023 +0800 [feat](stats) Enable auto sample by default (#25607) --- fe/fe-common/src/main/java/org/apache/doris/common/Config.java | 2 +- .../src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java index 72ecebca781..8fc192a6d94 100644 --- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java +++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java @@ -2164,7 +2164,7 @@ public class Config extends ConfigBase { @ConfField(description = {"是否开启大表自动sample,开启后对于大小超过huge_table_lower_bound_size_in_bytes会自动通过采样收集" + "统计信息", "Whether to enable automatic sampling for large tables, which, when enabled, automatically" + "collects statistics through sampling for tables larger than 'huge_table_lower_bound_size_in_bytes'"}) -public static boolean enable_auto_sample = false; +public static boolean enable_auto_sample = true; @ConfField(description = { "控制统计信息的自动触发作业执行记录的持久化行数", diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java index 04b775bcd63..81cc97a63bf 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java +++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java @@ -120,11 +120,11 @@ public abstract class BaseAnalysisTask { } protected void init(AnalysisInfo info) { -tableSample = getTableSample(); DBObjects dbObjects = StatisticsUtil.convertIdToObjects(info.catalogId, info.dbId, info.tblId); catalog = dbObjects.catalog; db = dbObjects.db; tbl = dbObjects.table; +tableSample = getTableSample(); // External Table level task doesn't contain a column. Don't need to do the column related analyze. if (info.externalTableLevelTask) { return; - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](stats) Enable auto sample by default [doris]
xiaokang merged PR #25607: URL: https://github.com/apache/doris/pull/25607 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](exec) Fix date dict infinite loop. [doris]
doris-robot commented on PR #25570: URL: https://github.com/apache/doris/pull/25570#issuecomment-1773858245 TeamCity be ut coverage result: Function Coverage: 37.04% (8301/22410) Line Coverage: 29.20% (66677/228330) Region Coverage: 27.83% (34607/124360) Branch Coverage: 24.42% (17583/71996) Coverage Report: http://coverage.selectdb-in.cc/coverage/c797a35110db93fb57bb560c4790bd257d8375a7_c797a35110db93fb57bb560c4790bd257d8375a7/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](exec) Fix date dict infinite loop. [doris]
doris-robot commented on PR #25570: URL: https://github.com/apache/doris/pull/25570#issuecomment-1773858496 TeamCity be ut coverage result: Function Coverage: 37.04% (8301/22410) Line Coverage: 29.20% (66675/228330) Region Coverage: 27.83% (34612/124360) Branch Coverage: 24.42% (17582/71996) Coverage Report: http://coverage.selectdb-in.cc/coverage/c797a35110db93fb57bb560c4790bd257d8375a7_c797a35110db93fb57bb560c4790bd257d8375a7/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](alter backends)backend modify & drop & decommission by ids [doris]
doris-robot commented on PR #25444: URL: https://github.com/apache/doris/pull/25444#issuecomment-1773860036 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.21 seconds stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.4 seconds inserted 1000 Rows, about 340K ops/s storage size: 17162340604 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](recover) support skipping missing version in select by session variable [doris]
xy720 commented on PR #25654: URL: https://github.com/apache/doris/pull/25654#issuecomment-1773861471 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](recover) support skipping missing version in select by session variable [doris]
github-actions[bot] commented on PR #25654: URL: https://github.com/apache/doris/pull/25654#issuecomment-1773863530 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](recover) support skipping missing version in select by session variable [doris]
github-actions[bot] commented on PR #25654: URL: https://github.com/apache/doris/pull/25654#issuecomment-1773867151 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](decimal) support decimal256 [doris]
jacktengg commented on PR #25386: URL: https://github.com/apache/doris/pull/25386#issuecomment-1773870984 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](decimal) support decimal256 [doris]
github-actions[bot] commented on code in PR #25386: URL: https://github.com/apache/doris/pull/25386#discussion_r1367772481 ## be/test/vec/data_types/decimal_test.cpp: ## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include +#include +#include + +#include +#include + +#include "gtest/gtest_pred_impl.h" +#include "runtime/raw_value.h" +#include "runtime/type_limit.h" +#include "util/string_parser.hpp" +#include "vec/core/types.h" +#include "vec/data_types/data_type_decimal.h" +namespace doris::vectorized { + +TEST(DecimalTest, Decimal256) { +// +Decimal256 dec1(type_limit::max()); +auto des_str = dec1.to_string(10); +EXPECT_EQ(des_str, + "99.99"); +des_str = dec1.to_string(0); +EXPECT_EQ(des_str, + ""); +des_str = dec1.to_string(76); +EXPECT_EQ(des_str, + "0."); + +auto dec2 = type_limit::min(); +des_str = dec2.to_string(10); +EXPECT_EQ(des_str, + "-99.99"); +des_str = dec2.to_string(0); +EXPECT_EQ(des_str, + "-"); +des_str = dec2.to_string(76); +EXPECT_EQ(des_str, + "-0."); + +// plus +Decimal256 dec3 = dec1 + dec2; +des_str = dec3.to_string(10); +EXPECT_EQ(des_str, "0.00"); +des_str = dec3.to_string(0); +EXPECT_EQ(des_str, "0"); +des_str = dec3.to_string(76); +EXPECT_EQ(des_str, + "0."); + +// minus +dec2 = type_limit::max(); +dec3 = dec1 - dec2; +des_str = dec3.to_string(10); +EXPECT_EQ(des_str, "0.00"); + +// multiply + +// divide +dec1 = type_limit::max(); +dec2 = vectorized::Decimal256(10); +dec3 = dec1 / dec2; +des_str = dec3.to_string(1); +EXPECT_EQ(des_str, + "99.9"); + +// overflow +} + +TEST(DecimalTest, compare) { Review Comment: warning: all parameters should be named in a function [readability-named-parameter] ```suggestion TEST(DecimalTest /*unused*/, compare /*unused*/) { ``` ## be/test/vec/data_types/decimal_test.cpp: ## @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include +#include +#include + +#include +#include + +#include "gtest/gtest_pred_impl.h" +#include "runtime/raw_value.h" +#include "runtime/type_limit.h" +#include "util/string_parser.hpp" +#include "vec/core/types.h" +#include "vec/data_types/data_type_decimal.h" +namespace doris::vectorized { + +TEST(DecimalTest, Decimal256) { +// +Decimal256 dec1(type_limit::max()); +auto des_str = dec1.to_string(10); +EXPECT_EQ(des_str, + "9
Re: [PR] [Fix](exec) Fix date dict infinite loop. [doris]
doris-robot commented on PR #25570: URL: https://github.com/apache/doris/pull/25570#issuecomment-1773877852 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.12 seconds stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.2 seconds inserted 1000 Rows, about 342K ops/s storage size: 17161905632 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](recover) support skipping missing version in select by session variable [doris]
doris-robot commented on PR #25654: URL: https://github.com/apache/doris/pull/25654#issuecomment-1773882417 TeamCity be ut coverage result: Function Coverage: 37.04% (8301/22411) Line Coverage: 29.20% (3/228334) Region Coverage: 27.83% (34611/124366) Branch Coverage: 24.41% (17578/72002) Coverage Report: http://coverage.selectdb-in.cc/coverage/f096b41aaf47ad5f30fbb86a151b0ac7963d841f_f096b41aaf47ad5f30fbb86a151b0ac7963d841f/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](hive)Support hive tables after alter type. [doris]
hubgeter commented on PR #25138: URL: https://github.com/apache/doris/pull/25138#issuecomment-1773890160 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](hive)Support hive tables after alter type. [doris]
github-actions[bot] commented on code in PR #25138: URL: https://github.com/apache/doris/pull/25138#discussion_r1367781725 ## be/src/vec/exec/format/parquet/vparquet_column_reader.cpp: ## @@ -476,86 +481,109 @@ Status ScalarColumnReader::_try_load_dict_page(bool* loaded, bool* has_dict) { Status ScalarColumnReader::read_column_data(ColumnPtr& doris_column, DataTypePtr& type, ColumnSelectVector& select_vector, size_t batch_size, size_t* read_rows, bool* eof, bool is_dict_filter) { -if (_chunk_reader->remaining_num_values() == 0) { -if (!_chunk_reader->has_next_page()) { -*eof = true; -*read_rows = 0; -return Status::OK(); -} -RETURN_IF_ERROR(_chunk_reader->next_page()); -} -if (_nested_column) { -RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent()); -return _read_nested_column(doris_column, type, select_vector, batch_size, read_rows, eof, - is_dict_filter); -} - -// generate the row ranges that should be read -std::list read_ranges; -_generate_read_ranges(_current_row_index, - _current_row_index + _chunk_reader->remaining_num_values(), read_ranges); -if (read_ranges.size() == 0) { -// skip the whole page -_current_row_index += _chunk_reader->remaining_num_values(); -RETURN_IF_ERROR(_chunk_reader->skip_page()); -*read_rows = 0; -} else { -bool skip_whole_batch = false; -// Determining whether to skip page or batch will increase the calculation time. -// When the filtering effect is greater than 60%, it is possible to skip the page or batch. -if (select_vector.has_filter() && select_vector.filter_ratio() > 0.6) { -// lazy read -size_t remaining_num_values = 0; -for (auto& range : read_ranges) { -remaining_num_values += range.last_row - range.first_row; -} -if (batch_size >= remaining_num_values && -select_vector.can_filter_all(remaining_num_values)) { -// We can skip the whole page if the remaining values is filtered by predicate columns -select_vector.skip(remaining_num_values); -_current_row_index += _chunk_reader->remaining_num_values(); -RETURN_IF_ERROR(_chunk_reader->skip_page()); -*read_rows = remaining_num_values; -if (!_chunk_reader->has_next_page()) { -*eof = true; -} +bool need_convert = false; +auto& parquet_physical_type = _chunk_meta.meta_data.type; +auto& show_type = _field_schema->type.type; + +ColumnPtr src_column = ParquetConvert::get_column(parquet_physical_type, show_type, + doris_column, type, &need_convert); + +do { +if (_chunk_reader->remaining_num_values() == 0) { +if (!_chunk_reader->has_next_page()) { +*eof = true; +*read_rows = 0; return Status::OK(); } -skip_whole_batch = -batch_size <= remaining_num_values && select_vector.can_filter_all(batch_size); -if (skip_whole_batch) { -select_vector.skip(batch_size); -} +RETURN_IF_ERROR(_chunk_reader->next_page()); } -// load page data to decode or skip values -RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent()); -size_t has_read = 0; -for (auto& range : read_ranges) { -// generate the skipped values -size_t skip_values = range.first_row - _current_row_index; -RETURN_IF_ERROR(_skip_values(skip_values)); -_current_row_index += skip_values; -// generate the read values -size_t read_values = -std::min((size_t)(range.last_row - range.first_row), batch_size - has_read); -if (skip_whole_batch) { -RETURN_IF_ERROR(_skip_values(read_values)); -} else { -RETURN_IF_ERROR(_read_values(read_values, doris_column, type, select_vector, - is_dict_filter)); +if (_nested_column) { +RETURN_IF_ERROR(_chunk_reader->load_page_data_idempotent()); +RETURN_IF_ERROR(_read_nested_column(src_column, type, select_vector, batch_size, +read_rows, eof, is_dict_filter)); +break; +} + +// generate the row ranges that should be read +std::list read_ranges; +_generate_read_ranges(_current_row_index, + _current_row_index + _chunk_reader->remaining_num_values(), +
Re: [PR] [feature](decimal) support decimal256 [doris]
doris-robot commented on PR #25386: URL: https://github.com/apache/doris/pull/25386#issuecomment-1773892872 TeamCity be ut coverage result: Function Coverage: 37.21% (8378/22513) Line Coverage: 29.37% (67536/229926) Region Coverage: 27.95% (35028/125340) Branch Coverage: 24.50% (17741/72414) Coverage Report: http://coverage.selectdb-in.cc/coverage/4d81ec09ed4ccaf74c79c7e700cdb27c2c004b90_4d81ec09ed4ccaf74c79c7e700cdb27c2c004b90/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](decimal) support decimal256 [doris]
doris-robot commented on PR #25386: URL: https://github.com/apache/doris/pull/25386#issuecomment-1773901653 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.89 seconds stream load tsv: 551 seconds loaded 74807831229 Bytes, about 129 MB/s stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.0 seconds inserted 1000 Rows, about 344K ops/s storage size: 17162042504 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](rf) fix minmax runtime filter [doris]
github-actions[bot] commented on PR #25720: URL: https://github.com/apache/doris/pull/25720#issuecomment-1773949947 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix] Fix the extension mysql_to_doris bug #18985 [doris]
github-actions[bot] commented on PR #18988: URL: https://github.com/apache/doris/pull/18988#issuecomment-1773951098 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancment](quantile_union): support quantile_union on nullable columns [doris]
github-actions[bot] commented on PR #17793: URL: https://github.com/apache/doris/pull/17793#issuecomment-177395 We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[doris] branch branch-2.0 updated (41fe511ae8d -> 243a5f52cdf)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a change to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git from 41fe511ae8d [feat](stats) Enable auto sample by default (#25607) new bd3ee3ebdca [feature](function) Support SHA family functions (#24342) new ef099419375 [Improvement](hive-udf)(doc) minimize hive-udf and add some docs. (#24786) new a92632700cd [Enhance](regression) add hive out file check (#25475) new 3dab722500f [cases](regresstest)add json format regress test for nested types (#25397) new 243a5f52cdf [Improvement](statistics)Collect stats for hive partition column using metadata (#24853) The 5 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/util/sha.cpp| 118 ++ be/src/util/sha.h | 75 ++ be/src/vec/functions/function_string.cpp | 11 +- be/src/vec/functions/function_string.h | 107 - docs/en/docs/ecosystem/hive-bitmap-udf.md | 56 - docs/en/docs/query-acceleration/statistics.md | 15 +- .../sql-functions/encrypt-digest-functions/sha.md | 53 + .../sql-functions/encrypt-digest-functions/sha2.md | 70 ++ docs/sidebars.json | 4 +- docs/zh-CN/docs/ecosystem/hive-bitmap-udf.md | 56 - docs/zh-CN/docs/query-acceleration/statistics.md | 184 +-- .../sql-functions/encrypt-digest-functions/sha.md | 54 + .../sql-functions/encrypt-digest-functions/sha2.md | 70 ++ fe/fe-core/src/main/cup/sql_parser.cup | 20 ++ .../apache/doris/analysis/AnalyzeProperties.java | 7 +- .../org/apache/doris/analysis/AnalyzeStmt.java | 4 + .../apache/doris/analysis/FunctionCallExpr.java| 13 ++ .../doris/catalog/BuiltinScalarFunctions.java | 4 + .../doris/datasource/hive/HiveMetaStoreCache.java | 14 +- .../apache/doris/external/hive/util/HiveUtil.java | 16 ++ .../trees/expressions/functions/scalar/Sha1.java | 68 ++ .../trees/expressions/functions/scalar/Sha2.java | 69 ++ .../expressions/visitor/ScalarFunctionVisitor.java | 10 + .../org/apache/doris/statistics/AnalysisInfo.java | 8 +- .../doris/statistics/AnalysisInfoBuilder.java | 20 +- .../apache/doris/statistics/AnalysisManager.java | 1 + .../apache/doris/statistics/BaseAnalysisTask.java | 18 ++ .../apache/doris/statistics/HMSAnalysisTask.java | 154 +++-- .../doris/statistics/util/StatisticsUtil.java | 5 + fe/fe-core/src/main/jflex/sql_scanner.flex | 1 + fe/hive-udf/pom.xml| 75 -- gensrc/script/doris_builtins_functions.py | 7 + .../test_nestedtypes_json_insert_into_with_s3.out | 18 +- .../hive/test_prepare_hive_data_in_case.out| 11 + .../encryption_digest/test_digest.out | 39 .../sql_functions/encryption_digest/test_md5.out | 6 - .../org/apache/doris/regression/suite/Suite.groovy | 48 +++- ...est_nestedtypes_json_insert_into_with_s3.groovy | 253 + .../hive/test_prepare_hive_data_in_case.groovy | 11 +- .../hive/test_hive_partition_column_analyze.groovy | 230 +++ .../{test_md5.groovy => test_digest.groovy}| 19 +- 41 files changed, 1737 insertions(+), 285 deletions(-) create mode 100644 be/src/util/sha.cpp create mode 100644 be/src/util/sha.h create mode 100644 docs/en/docs/sql-manual/sql-functions/encrypt-digest-functions/sha.md create mode 100644 docs/en/docs/sql-manual/sql-functions/encrypt-digest-functions/sha2.md create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/encrypt-digest-functions/sha.md create mode 100644 docs/zh-CN/docs/sql-manual/sql-functions/encrypt-digest-functions/sha2.md create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Sha1.java create mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/Sha2.java create mode 100644 regression-test/data/external_table_p0/hive/test_prepare_hive_data_in_case.out create mode 100644 regression-test/data/query_p0/sql_functions/encryption_digest/test_digest.out delete mode 100644 regression-test/data/query_p0/sql_functions/encryption_digest/test_md5.out create mode 100644 regression-test/suites/datatype_p0/nested_types/query/test_nestedtypes_json_insert_into_with_s3.groovy create mode 100644 regression-test/suites/external_table_p2/hive/test_hive_partition_column_analyze.groovy rename regression-test/suites/query_p0/sql_functions/encryption_digest/{test_md5.groovy => test_digest.groovy} (53%) ---
[doris] 04/05: [cases](regresstest)add json format regress test for nested types (#25397)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git commit 3dab722500f3062e0938d88d019736cd579bbe10 Author: amory AuthorDate: Tue Oct 17 10:16:52 2023 +0800 [cases](regresstest)add json format regress test for nested types (#25397) --- .../test_nestedtypes_json_insert_into_with_s3.out | 18 +- ...est_nestedtypes_json_insert_into_with_s3.groovy | 253 + 2 files changed, 262 insertions(+), 9 deletions(-) diff --git a/regression-test/data/datatype_p0/nested_types/query/test_nestedtypes_json_insert_into_with_s3.out b/regression-test/data/datatype_p0/nested_types/query/test_nestedtypes_json_insert_into_with_s3.out index 21fb51ac176..a125a032e3c 100644 --- a/regression-test/data/datatype_p0/nested_types/query/test_nestedtypes_json_insert_into_with_s3.out +++ b/regression-test/data/datatype_p0/nested_types/query/test_nestedtypes_json_insert_into_with_s3.out @@ -1,33 +1,33 @@ -- This file is automatically generated. You should know what you did if you want to edit this -- !sql_arr_json_without_quote_s3 -- -0 [false,true,false,true,true,true,false,false,false,false,false,true,true,true,false,false,true,true,true,false,false,false,false,true,false,true,true,false,true,false,true,false,true,false,true,true,false,false,true,true,false,false,true,true,false,true,false,true,true,false,false,true,true,false,false,true,true,true,true,false,false,false,false,false,false,false,true,true,true,true,false,true,false,true,true,true,true,false,false,false,true,true,true,false,true,false,false,false,false [...] +0 [false,true,false,true,true,true,false,false,false,false,false,true,true,true,false,false,true,true,true,false,false,false,false,true,false,true,true,false,true,false,true,false,true,false,true,true,false,false,true,true,false,false,true,true,false,true,false,true,true,false,false,true,true,false,false,true,true,true,true,false,false,false,false,false,false,false,true,true,true,true,false,true,false,true,true,true,true,false,false,false,true,true,true,false,true,false,false,false,false [...] -- !sql_arr_json_without_quote_doris -- -0 [0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1] [NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NU [...] +0 [0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1] [NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NU [...] -- !sql_arr_json_without_quote_s3 -- -0 [[false,true],[true,true],[false,false],[],[false],[false],[],[],[],[false,false],[],[],[true],[true],[false],[true],[true],[],[true,false],[],[],[false],[false],[],[true,false],[false,true],[],[false,false],[true,false],[false,true],[false,false],[],[],[true],[true],[true],[],[],[false,true],[false,false],[],[],[true,false],[false],[false],[true],[],[],[],[true],[false,false],[true,false],[false,false],[false],[false,true],[],[false],[false,true],[false,false],[],[false],[true],[],[], [...] +0 [[false,true],[true,true],[false,false],[],[false],[false],[],[],[],[false,false],[],[],[true],[true],[false],[true],[true],[],[true,false],[],[],[false],[false],[],[true,false],[false,true],[],[false,false],[true,false],[false,true],[false,false],[],[],[true],[true],[true],[],[],[false,true],[false,false],[],[],[true,false],[false],[false],[true],[],[],[],[true],[false,false],[true,false],[false,false],[false],[false,true],[],[false],[false,true],[false,false],[],[false],[true],[],[], [...] -- !sql_arr_json_without_quote_doris -- -0 [[0, 1], [1, 1], [0, 0], [], [0], [0], [], [], [], [0, 0], [], [], [1], [1], [0], [1], [1], [], [1, 0], [], [], [0], [0], [], [1, 0], [0, 1], [], [0, 0], [1, 0], [0, 1], [0, 0], [], [], [1], [1], [1], [], [], [0, 1], [0, 0], [], [], [1, 0], [0], [0], [1], [], [], [], [1], [0, 0], [1, 0], [0, 0], [0], [0, 1], [], [0], [0, 1], [0, 0], [], [0], [1], [], [], [], [1, 0], [], [], [1], [1], [], [], [1, 0], [1, 0], [0, 0], [], [], [0, 0], [0, 1], [1], [1], [1], [1, 0], [1], [], [0, 0], [], [0, [...] +0 [[0, 1], [1, 1], [0, 0], [], [0], [0], [], [], [], [0, 0], [], [], [1], [1], [0], [1], [1], [], [1, 0], [], [], [0], [0], [],
[doris] 01/05: [feature](function) Support SHA family functions (#24342)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git commit bd3ee3ebdca08743c14ac62f971929f7d98aee68 Author: zclllyybb AuthorDate: Wed Sep 20 17:21:45 2023 +0800 [feature](function) Support SHA family functions (#24342) --- be/src/util/sha.cpp| 118 + be/src/util/sha.h | 75 + be/src/vec/functions/function_string.cpp | 11 +- be/src/vec/functions/function_string.h | 107 ++- .../sql-functions/encrypt-digest-functions/sha.md | 53 + .../sql-functions/encrypt-digest-functions/sha2.md | 70 docs/sidebars.json | 4 +- .../sql-functions/encrypt-digest-functions/sha.md | 54 ++ .../sql-functions/encrypt-digest-functions/sha2.md | 70 .../apache/doris/analysis/FunctionCallExpr.java| 13 +++ .../doris/catalog/BuiltinScalarFunctions.java | 4 + .../trees/expressions/functions/scalar/Sha1.java | 68 .../trees/expressions/functions/scalar/Sha2.java | 69 .../expressions/visitor/ScalarFunctionVisitor.java | 10 ++ gensrc/script/doris_builtins_functions.py | 7 ++ .../encryption_digest/test_digest.out | 39 +++ .../sql_functions/encryption_digest/test_md5.out | 6 -- .../{test_md5.groovy => test_digest.groovy}| 19 +++- 18 files changed, 781 insertions(+), 16 deletions(-) diff --git a/be/src/util/sha.cpp b/be/src/util/sha.cpp new file mode 100644 index 000..68099ff9269 --- /dev/null +++ b/be/src/util/sha.cpp @@ -0,0 +1,118 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#include "util/sha.h" + +#include + +#include + +namespace doris { + +constexpr static char dig_vec_lower[] = "0123456789abcdef"; + +void SHA1Digest::reset(const void* data, size_t length) { +SHA1_Init(&_sha_ctx); +SHA1_Update(&_sha_ctx, data, length); +} + +std::string_view SHA1Digest::digest() { +unsigned char buf[SHA_DIGEST_LENGTH]; +SHA1_Final(buf, &_sha_ctx); + +char* to = _reuse_hex; +for (int i = 0; i < SHA_DIGEST_LENGTH; ++i) { +*to++ = dig_vec_lower[buf[i] >> 4]; +*to++ = dig_vec_lower[buf[i] & 0x0F]; +} + +return std::string_view {_reuse_hex, _reuse_hex + 2 * SHA_DIGEST_LENGTH}; +} + +void SHA224Digest::reset(const void* data, size_t length) { +SHA224_Init(&_sha224_ctx); +SHA224_Update(&_sha224_ctx, data, length); +} + +std::string_view SHA224Digest::digest() { +unsigned char buf[SHA224_DIGEST_LENGTH]; +SHA224_Final(buf, &_sha224_ctx); + +char* to = _reuse_hex; +for (int i = 0; i < SHA224_DIGEST_LENGTH; ++i) { +*to++ = dig_vec_lower[buf[i] >> 4]; +*to++ = dig_vec_lower[buf[i] & 0x0F]; +} + +return std::string_view {_reuse_hex, _reuse_hex + 2 * SHA224_DIGEST_LENGTH}; +} + +void SHA256Digest::reset(const void* data, size_t length) { +SHA256_Init(&_sha256_ctx); +SHA256_Update(&_sha256_ctx, data, length); +} + +std::string_view SHA256Digest::digest() { +unsigned char buf[SHA256_DIGEST_LENGTH]; +SHA256_Final(buf, &_sha256_ctx); + +char* to = _reuse_hex; +for (int i = 0; i < SHA256_DIGEST_LENGTH; ++i) { +*to++ = dig_vec_lower[buf[i] >> 4]; +*to++ = dig_vec_lower[buf[i] & 0x0F]; +} + +return std::string_view {_reuse_hex, _reuse_hex + 2 * SHA256_DIGEST_LENGTH}; +} + +void SHA384Digest::reset(const void* data, size_t length) { +SHA384_Init(&_sha384_ctx); +SHA384_Update(&_sha384_ctx, data, length); +} + +std::string_view SHA384Digest::digest() { +unsigned char buf[SHA384_DIGEST_LENGTH]; +SHA384_Final(buf, &_sha384_ctx); + +char* to = _reuse_hex; +for (int i = 0; i < SHA384_DIGEST_LENGTH; ++i) { +*to++ = dig_vec_lower[buf[i] >> 4]; +*to++ = dig_vec_lower[buf[i] & 0x0F]; +} + +return std::string_view {_reuse_hex, _reuse_hex + 2 * SHA384_DIGEST_LENGTH}; +} + +void SHA512Digest::reset(const void* data, size_t length) { +SHA512_Init(&_sha512_ctx); +SHA512_Update(&
[doris] 03/05: [Enhance](regression) add hive out file check (#25475)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git commit a92632700cdd32481e8acbf7ca4f81e7cdc922a6 Author: zhangguoqiang <18372634...@163.com> AuthorDate: Tue Oct 17 10:11:57 2023 +0800 [Enhance](regression) add hive out file check (#25475) add hive out file check fix hive sql state with " ; " --- .../hive/test_prepare_hive_data_in_case.out| 11 + .../org/apache/doris/regression/suite/Suite.groovy | 48 +++--- .../hive/test_prepare_hive_data_in_case.groovy | 11 - 3 files changed, 62 insertions(+), 8 deletions(-) diff --git a/regression-test/data/external_table_p0/hive/test_prepare_hive_data_in_case.out b/regression-test/data/external_table_p0/hive/test_prepare_hive_data_in_case.out new file mode 100644 index 000..7259a57a915 --- /dev/null +++ b/regression-test/data/external_table_p0/hive/test_prepare_hive_data_in_case.out @@ -0,0 +1,11 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !hive_docker_01 -- +eeefff +cccddd +aaabbb + +-- !sql_02 -- +eeefff +cccddd +aaabbb + diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy index 6cd22bb663f..f03a59a0042 100644 --- a/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy +++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/suite/Suite.groovy @@ -551,7 +551,7 @@ class Suite implements GroovyInterceptable { } List> hive_docker(String sqlStr, boolean isOrder = false){ -String cleanedSqlStr = sqlStr.replaceAll(/;+$/, '') +String cleanedSqlStr = sqlStr.replaceAll("\\s*;\\s*\$", "") def (result, meta) = JdbcUtils.executeToList(context.getHiveDockerConnection(), cleanedSqlStr) if (isOrder) { result = DataUtils.sortByToString(result) @@ -560,7 +560,7 @@ class Suite implements GroovyInterceptable { } List> hive_remote(String sqlStr, boolean isOrder = false){ -String cleanedSqlStr = sqlStr.replaceAll(/;+$/, '') +String cleanedSqlStr = sqlStr.replaceAll("\\s*;\\s*\$", "") def (result, meta) = JdbcUtils.executeToList(context.getHiveRemoteConnection(), cleanedSqlStr) if (isOrder) { result = DataUtils.sortByToString(result) @@ -572,9 +572,23 @@ class Suite implements GroovyInterceptable { if (context.config.generateOutputFile || context.config.forceGenerateOutputFile) { Tuple2>, ResultSetMetaData> tupleResult = null if (arg instanceof PreparedStatement) { -tupleResult = JdbcUtils.executeToStringList(context.getConnection(), (PreparedStatement) arg) +if (tag.contains("hive_docker")) { +tupleResult = JdbcUtils.executeToStringList(context.getHiveDockerConnection(), (PreparedStatement) arg) +}else if (tag.contains("hive_remote")) { +tupleResult = JdbcUtils.executeToStringList(context.getHiveRemoteConnection(), (PreparedStatement) arg) +} +else{ +tupleResult = JdbcUtils.executeToStringList(context.getConnection(), (PreparedStatement) arg) +} } else { -tupleResult = JdbcUtils.executeToStringList(context.getConnection(), (String) arg) +if (tag.contains("hive_docker")) { +tupleResult = JdbcUtils.executeToStringList(context.getHiveDockerConnection(), (String) arg) +}else if (tag.contains("hive_remote")) { +tupleResult = JdbcUtils.executeToStringList(context.getHiveRemoteConnection(), (String) arg) +} +else{ +tupleResult = JdbcUtils.executeToStringList(context.getConnection(), (String) arg) +} } def (result, meta) = tupleResult if (isOrder) { @@ -596,9 +610,23 @@ class Suite implements GroovyInterceptable { OutputUtils.TagBlockIterator expectCsvResults = context.getOutputIterator().next() Tuple2>, ResultSetMetaData> tupleResult = null if (arg instanceof PreparedStatement) { -tupleResult = JdbcUtils.executeToStringList(context.getConnection(), (PreparedStatement) arg) +if (tag.contains("hive_docker")) { +tupleResult = JdbcUtils.executeToStringList(context.getHiveDockerConnection(), (PreparedStatement) arg) +}else if (tag.contains("hive_remote")) { +tupleResult = JdbcUtils.executeToStringList(context.getHiveRemoteConnection(), (PreparedStatement) arg) +
[doris] 05/05: [Improvement](statistics)Collect stats for hive partition column using metadata (#24853)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git commit 243a5f52cdf8698d72aac242fb2150288ab52185 Author: Jibing-Li <64681310+jibing...@users.noreply.github.com> AuthorDate: Tue Oct 17 10:31:57 2023 +0800 [Improvement](statistics)Collect stats for hive partition column using metadata (#24853) Hive partition columns' stats could be calculated from hive metastore data. Doesn't need to execute sql to get the stats. This PR is using hive partition metadata to collect partition column stats. --- docs/en/docs/query-acceleration/statistics.md | 15 +- docs/zh-CN/docs/query-acceleration/statistics.md | 184 + fe/fe-core/src/main/cup/sql_parser.cup | 20 ++ .../apache/doris/analysis/AnalyzeProperties.java | 7 +- .../org/apache/doris/analysis/AnalyzeStmt.java | 4 + .../doris/datasource/hive/HiveMetaStoreCache.java | 14 +- .../apache/doris/external/hive/util/HiveUtil.java | 16 ++ .../org/apache/doris/statistics/AnalysisInfo.java | 8 +- .../doris/statistics/AnalysisInfoBuilder.java | 20 +- .../apache/doris/statistics/AnalysisManager.java | 1 + .../apache/doris/statistics/BaseAnalysisTask.java | 18 ++ .../apache/doris/statistics/HMSAnalysisTask.java | 154 +++--- .../doris/statistics/util/StatisticsUtil.java | 5 + fe/fe-core/src/main/jflex/sql_scanner.flex | 1 + .../hive/test_hive_partition_column_analyze.groovy | 230 + 15 files changed, 469 insertions(+), 228 deletions(-) diff --git a/docs/en/docs/query-acceleration/statistics.md b/docs/en/docs/query-acceleration/statistics.md index 2ab48ebc573..28795d01deb 100644 --- a/docs/en/docs/query-acceleration/statistics.md +++ b/docs/en/docs/query-acceleration/statistics.md @@ -79,8 +79,11 @@ The user triggers a manual collection job through a statement `ANALYZE` to colle Column statistics collection syntax: ```SQL -ANALYZE TABLE | DATABASE table_name | db_name -[ (column_name [, ...]) ][ [ WITH SYNC ] [ WITH INCREMENTAL ] [ WITH SAMPLE PERCENT | ROWS ] [ WITH PERIOD ] [WITH HISTOGRAM]][ PROPERTIES ("key" = "value", ...) ]; +ANALYZE < TABLE | DATABASE table_name | db_name > +[ PARTITIONS [(*) | (partition_name [, ...]) | WITH RECENT COUNT ] ] +[ (column_name [, ...]) ] +[ [ WITH SYNC ] [ WITH SAMPLE PERCENT | ROWS ] [ WITH SQL ] ] +[ PROPERTIES ("key" = "value", ...) ]; ``` Explanation: @@ -121,8 +124,12 @@ mysql -uroot -P9030 -h192.168.xxx.xxx``` Create a data table: -```SQL -mysql> CREATE DATABASE IF NOT EXISTS stats_test; +- `table_name`: Specifies the target table. It can be in the `db_name.table_name` format. +- `partition_name`: The specified target partitions(for hive external table only)。Must be partitions exist in `table_name`. Multiple partition names are separated by commas. e.g. for single level partition: PARTITIONS(`event_date=20230706`), for multi level partition: PARTITIONS(`nation=US/city=Washington`). PARTITIONS(*) specifies all partitions, PARTITIONS WITH RECENT 30 specifies the latest 30 partitions. +- `column_name`: Specifies the target column. It must be an existing column in `table_name`, and multiple column names are separated by commas. +- `sync`: Collect statistics synchronously. Returns upon completion. If not specified, it executes asynchronously and returns a task ID. +- `sample percent | rows`: Collect statistics using sampling. You can specify either the sampling percentage or the number of sampled rows. +- `sql`: Collect statistics for external partition column with sql. By default, it uses meta data for partition columns, which is faster but may inaccurate for row count and size. Using sql could collect the accurate stats. mysql> CREATE TABLE IF NOT EXISTS stats_test.example_tbl ( `user_id` LARGEINT NOT NULL,`date` DATEV2 NOT NULL, `city` VARCHAR(20),`age` SMALLINT,`sex` TINYINT, `last_visit_date` DATETIME REPLACE,`cost` BIGINT SUM, `max_dwell_time` INT MAX,`min_dwell_time` INT MIN) ENGINE=OLAP AGGREGATE KEY(`user_id`, `date`, `city`, `age`, `sex`)PARTITION BY LIST(`date`)(PARTITION `p_201701` VALUES IN ("2017-10-01"), PARTITION `p_201702` VALUES IN ("2017-10-02"), [...] diff --git a/docs/zh-CN/docs/query-acceleration/statistics.md b/docs/zh-CN/docs/query-acceleration/statistics.md index 2d893a39fc1..0362f989c1e 100644 --- a/docs/zh-CN/docs/query-acceleration/statistics.md +++ b/docs/zh-CN/docs/query-acceleration/statistics.md @@ -79,9 +79,10 @@ Doris 查询优化器使用统计信息来确定查询最有效的执行计划 列统计信息收集语法: ```SQL -ANALYZE TABLE | DATABASE table_name | db_name +ANALYZE < TABLE | DATABASE table_name | db_name > +[ PARTITIONS [(*) | (partition_name [, ...]) | WITH RECENT COUNT ] ] [ (column_name [, ...]) ] -[ [ WITH SYN
[doris] 02/05: [Improvement](hive-udf)(doc) minimize hive-udf and add some docs. (#24786)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git commit ef0994193758523aec0eb3398e016ec9b2bea7c6 Author: yagagagaga AuthorDate: Mon Oct 16 16:47:21 2023 +0800 [Improvement](hive-udf)(doc) minimize hive-udf and add some docs. (#24786) --- docs/en/docs/ecosystem/hive-bitmap-udf.md| 56 - docs/zh-CN/docs/ecosystem/hive-bitmap-udf.md | 56 - fe/hive-udf/pom.xml | 75 3 files changed, 163 insertions(+), 24 deletions(-) diff --git a/docs/en/docs/ecosystem/hive-bitmap-udf.md b/docs/en/docs/ecosystem/hive-bitmap-udf.md index 6629b9d6a24..7bb93bc90ba 100644 --- a/docs/en/docs/ecosystem/hive-bitmap-udf.md +++ b/docs/en/docs/ecosystem/hive-bitmap-udf.md @@ -72,7 +72,7 @@ mvn package -Dmaven.test.skip=true --You can also just package the hive-udf module mvn package -pl hive-udf -am -Dmaven.test.skip=true ``` -After packaging and compiling, enter the hive-udf directory and there will be a target directory,There will be hive-udf-jar-with-dependencies.jar package +After packaging and compiling, enter the hive-udf directory and there will be a target directory,There will be hive-udf.jar package ```sql -- Load the Hive Bitmap Udf jar package (Upload the compiled hive-udf jar package to HDFS) @@ -108,4 +108,58 @@ select k1,bitmap_union(uuid) from hive_bitmap_table group by k1 ## Hive Bitmap import into Doris + + +### Method 1:Catalog (recommended) + + + +When create a Hive table in the format specified as TEXT, for Binary type, Hive will be saved as a bash64 encoded string. Therefore, the binary data can be directly saved as Bitmap through bitmap_from_base64 function by using Doris's Hive Catalog. + +Here is a full example: + +1. Creating Hive Tables in Hive + +```sql +CREATE TABLE IF NOT EXISTS `test`.`hive_bitmap_table`( +`k1` int COMMENT '', +`k2` StringCOMMENT '', +`k3` StringCOMMENT '', +`uuid` binaryCOMMENT 'bitmap' +) stored as textfile +``` + +2. [Creating a Catalog in Doris](../lakehouse/multi-catalog/hive) + +```sql +CREATE CATALOG hive PROPERTIES ( +'type'='hms', +'hive.metastore.uris' = 'thrift://127.0.0.1:9083' +); +``` + +3. Create Doris internal table + +```sql +CREATE TABLE IF NOT EXISTS `test`.`doris_bitmap_table`( +`k1` int COMMENT '', +`k2` StringCOMMENT '', +`k3` StringCOMMENT '', +`uuid` BITMAP BITMAP_UNION COMMENT 'bitmap' +) +AGGREGATE KEY(k1, k2, k3) +DISTRIBUTED BY HASH(`user_id`) BUCKETS 1 +PROPERTIES ( +"replication_allocation" = "tag.location.default: 1" +); +``` + +4. Inserting data from Hive into Doris + +```sql +insert into doris_bitmap_table select k1, k2, k3, bitmap_from_base64(uuid) from hive.test.hive_bitmap_table; +``` + +### Method 2:Spark Load + see details: [Spark Load](../data-operate/import/import-way/spark-load-manual.md) -> Basic operation -> Create load(Example 3: when the upstream data source is hive binary type table) diff --git a/docs/zh-CN/docs/ecosystem/hive-bitmap-udf.md b/docs/zh-CN/docs/ecosystem/hive-bitmap-udf.md index 7f886352ed4..d10a9250283 100644 --- a/docs/zh-CN/docs/ecosystem/hive-bitmap-udf.md +++ b/docs/zh-CN/docs/ecosystem/hive-bitmap-udf.md @@ -74,7 +74,7 @@ mvn package -Dmaven.test.skip=true --也可以只打hive-udf module mvn package -pl hive-udf -am -Dmaven.test.skip=true ``` -打包编译完成进入hive-udf目录会有target目录,里面就会有打包完成的hive-udf-jar-with-dependencies.jar包 +打包编译完成进入hive-udf目录会有target目录,里面就会有打包完成的hive-udf.jar包 ```sql @@ -117,4 +117,58 @@ select k1,bitmap_union(uuid) from hive_bitmap_table group by k1 ## Hive bitmap 导入 doris + + +### 方法一:Catalog (推荐) + + + +创建 Hive 表指定为 TEXT 格式,此时,对于 Binary 类型,Hive 会以 bash64 编码的字符串形式保存,此时可以通过 Hive Catalog 的形式,直接将位图数据通过 bitmap_from_bash64 函数插入到 Doris 内部。 + +以下是一个完整的例子: + +1. 在 Hive 中创建 Hive 表 + +```sql +CREATE TABLE IF NOT EXISTS `test`.`hive_bitmap_table`( +`k1` int COMMENT '', +`k2` StringCOMMENT '', +`k3` StringCOMMENT '', +`uuid` binaryCOMMENT 'bitmap' +) stored as textfile +``` + +2. [在 Doris 中创建 Catalog](../lakehouse/multi-catalog/hive) + +```sql +CREATE CATALOG hive PROPERTIES ( +'type'='hms', +'hive.metastore.uris' = 'thrift://127.0.0.1:9083' +); +``` + +3. 创建 Doris 内表 + +```sql +CREATE TABLE IF NOT EXISTS `test`.`doris_bitmap_table`( +`k1` int COMMENT '', +`k2` StringCOMMENT '', +`k3` StringCOMMENT '', +`uuid` BITMAP BITMAP_UNION COMMENT 'bitmap' +) +AGGREGATE KEY(k1, k2, k3) +DISTRIBUTED BY HASH(`user_id`) BUCKETS 1 +PROPERTIES ( +"replication_allocation" = "tag.location.default: 1" +); +``` + +4. 从 Hive 插入数据到 Doris 中 + +```sql +insert into doris_bitmap_table select k1, k2, k3, bitmap_from_base64(uuid) from hive.test.hive_bitmap_table; +``` + +##
Re: [PR] [feature](alter backends)backend modify & drop & decommission by ids [doris]
liugddx commented on PR #25444: URL: https://github.com/apache/doris/pull/25444#issuecomment-1773982701 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](CANCEL-ALTER-SYSTEM)decommission backend by ids [doris]
liugddx commented on PR #25441: URL: https://github.com/apache/doris/pull/25441#issuecomment-1773983098 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](decimal) support decimal256 [doris]
jacktengg commented on PR #25386: URL: https://github.com/apache/doris/pull/25386#issuecomment-1773983159 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](alter backends)backend modify & drop & decommission by ids [doris]
liugddx commented on PR #25444: URL: https://github.com/apache/doris/pull/25444#issuecomment-1773985417 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](CANCEL-ALTER-SYSTEM)decommission backend by ids [doris]
doris-robot commented on PR #25441: URL: https://github.com/apache/doris/pull/25441#issuecomment-1773989064 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 45.88 seconds stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 29.0 seconds inserted 1000 Rows, about 344K ops/s storage size: 17162008239 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](alter backends)backend modify & drop & decommission by ids [doris]
doris-robot commented on PR #25444: URL: https://github.com/apache/doris/pull/25444#issuecomment-1773990029 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.8 seconds stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.0 seconds inserted 1000 Rows, about 344K ops/s storage size: 17162108587 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](code-style)add spotless plugin and remove checkstyle [doris]
liugddx commented on PR #25033: URL: https://github.com/apache/doris/pull/25033#issuecomment-1773997988 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](function) support ip function ipv4stringtonum(ordefault, ornull), inet_aton [doris]
github-actions[bot] commented on PR #25510: URL: https://github.com/apache/doris/pull/25510#issuecomment-1774000468 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvements](status) catch some error status [doris]
BiteThet commented on PR #25677: URL: https://github.com/apache/doris/pull/25677#issuecomment-1774002400 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](recover) support skipping missing version in select by session variable [doris]
xy720 commented on PR #25654: URL: https://github.com/apache/doris/pull/25654#issuecomment-1774002887 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org