[PR] [fix] add null check before accessing pointer [doris]
Vallishp opened a new pull request, #38817: URL: https://github.com/apache/doris/pull/38817 ## Proposed changes Issue Number: close #xxx re2_ptr can possible null from code review. fixed code to handle it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix] add null check before accessing pointer [doris]
doris-robot commented on PR #38817: URL: https://github.com/apache/doris/pull/38817#issuecomment-2266633532 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix] add null check before accessing pointer [doris]
Vallishp commented on PR #38817: URL: https://github.com/apache/doris/pull/38817#issuecomment-2266633573 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix] add null check before accessing pointer [doris]
github-actions[bot] commented on PR #38817: URL: https://github.com/apache/doris/pull/38817#issuecomment-2266635340 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] fe: the part of code optimization in JdkUtils.java [doris]
doris-robot commented on PR #38818: URL: https://github.com/apache/doris/pull/38818#issuecomment-2266635910 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] fe: the part of code optimization in JdkUtils.java [doris]
mixinju opened a new pull request, #38818: URL: https://github.com/apache/doris/pull/38818 ## Proposed changes Issue Number: close #xxx 1. Returns the substring() result directly (直接返回substring()结果,不用中间变量) 2. Use Integer.parseInt() instead of Integer.valueOf(), This return value type is simply int -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](case) fix testcase out file upper case [doris]
xiaokang commented on PR #38815: URL: https://github.com/apache/doris/pull/38815#issuecomment-2266640189 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix] add null check before accessing pointer [doris]
doris-robot commented on PR #38817: URL: https://github.com/apache/doris/pull/38817#issuecomment-2266644042 TPC-H: Total hot run time: 41351 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 46f998a361643567eda0a45c8f55e886bc75de92, data reload: false -- Round 1 -- q1 17727 403640234023 q2 2021196 195 195 q3 10460 129513241295 q4 10182 806 865 806 q5 7575296629362936 q6 219 134 135 134 q7 1018610 602 602 q8 9426185719221857 q9 8693658765606560 q10 8732382838453828 q11 431 244 255 244 q12 433 227 223 223 q13 17768 294629852946 q14 268 247 244 244 q15 535 479 494 479 q16 534 392 388 388 q17 961 907 886 886 q18 7911732171497149 q19 1373122112041204 q20 557 324 341 324 q21 5291475547574755 q22 349 277 273 273 Total cold run time: 112464 ms Total hot run time: 41351 ms - Round 2, with runtime_filter_mode=off - q1 4069400241044002 q2 332 228 220 220 q3 2985297231262972 q4 1959196819621962 q5 5551547353975397 q6 226 131 129 129 q7 2115176118641761 q8 3319334133323332 q9 8611866487728664 q10 3963401538863886 q11 560 467 466 466 q12 782 620 607 607 q13 16721 309931273099 q14 320 270 272 270 q15 529 508 513 508 q16 460 423 414 414 q17 1739171217151712 q18 8239774476027602 q19 3586173516951695 q20 2100185818681858 q21 5680525654545256 q22 523 476 478 476 Total cold run time: 74369 ms Total hot run time: 56288 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix] add null check before accessing pointer [doris]
doris-robot commented on PR #38817: URL: https://github.com/apache/doris/pull/38817#issuecomment-2266646529 TPC-DS: Total hot run time: 168318 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 46f998a361643567eda0a45c8f55e886bc75de92, data reload: false query1 911 378 369 369 query2 6474167916191619 query3 6665215 221 215 query4 20138 17600 17267 17267 query5 3633513 517 513 query6 262 194 164 164 query7 4594292 296 292 query8 244 196 192 192 query9 8492231923122312 query10 436 275 283 275 query11 10454 10034 99179917 query12 119 90 87 87 query13 1622395 387 387 query14 9408645373736453 query15 209 160 163 160 query16 6977468 436 436 query17 944 561 553 553 query18 1920289 316 289 query19 184 145 148 145 query20 92 86 83 83 query21 204 100 99 99 query22 4200414639283928 query23 33725 33668 33443 33443 query24 10321 311931183118 query25 667 399 409 399 query26 1685152 157 152 query27 3053273 284 273 query28 7612201619631963 query29 1235437 438 437 query30 244 161 149 149 query31 914 749 778 749 query32 104 53 58 53 query33 670 312 323 312 query34 917 495 526 495 query35 884 764 766 764 query36 1036898 876 876 query37 286 81 82 81 query38 2956287028012801 query39 849 808 796 796 query40 249 127 112 112 query41 46 44 45 44 query42 118 107 97 97 query43 460 416 427 416 query44 1191738 737 737 query45 208 181 174 174 query46 1067823 785 785 query47 1794170317051703 query48 368 291 296 291 query49 951 419 417 417 query50 898 434 441 434 query51 6832655466906554 query52 105 88 91 88 query53 259 182 183 182 query54 617 446 449 446 query55 74 77 75 75 query56 270 246 253 246 query57 1148105010371037 query58 271 256 290 256 query59 2475223922582239 query60 285 270 275 270 query61 94 109 94 94 query62 922 645 670 645 query63 213 182 179 179 query64 5617190718931893 query65 3147309230863086 query66 1315342 330 330 query67 15131 14726 14838 14726 query68 4311557 566 557 query69 445 294 319 294 query70 1115112310851085 query71 419 283 290 283 query72 6998266425122512 query73 753 331 335 331 query74 6070567656495649 query75 3320268926792679 query76 2294119312611193 query77 410 308 304 304 query78 9508885988698859 query79 1920529 536 529 query80 1153518 498 498 query81 567 226 226 226 query82 1029132 133 132 query83 237 180 167 167 query84 260 79 77 77 query85 1283317 292 292 query86 463 309 313 309 query87 3294308530623062 query88 3005257925142514 query89 388 293 289 289 query90 1730192 194 192 query91 122 98 100 98 query92 58 49 50 49 query93 1806613 608 608 query94 843 292 292 292 query95 384 269 273 269 query96 608 282 280 280 query97 3277305330413041 query98 221 197 202 197 query99 1671127912961279 Total cold run time: 261427 ms Total hot run time: 168318 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s
Re: [PR] [fix] add null check before accessing pointer [doris]
doris-robot commented on PR #38817: URL: https://github.com/apache/doris/pull/38817#issuecomment-2266648156 ClickBench: Total hot run time: 30.79 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 46f998a361643567eda0a45c8f55e886bc75de92, data reload: false query1 0.040.040.04 query2 0.070.040.03 query3 0.230.040.05 query4 1.680.060.06 query5 0.470.480.48 query6 1.130.720.71 query7 0.020.020.01 query8 0.050.040.04 query9 0.560.510.52 query10 0.560.550.56 query11 0.160.120.12 query12 0.150.130.12 query13 0.610.610.60 query14 0.770.780.80 query15 0.900.860.85 query16 0.350.370.37 query17 1.001.001.01 query18 0.210.210.22 query19 1.881.721.76 query20 0.010.010.01 query21 15.41 0.760.67 query22 3.636.282.10 query23 17.84 1.441.36 query24 2.310.220.22 query25 0.180.080.08 query26 0.310.200.20 query27 0.460.220.23 query28 13.17 1.010.97 query29 12.58 3.303.27 query30 0.260.060.05 query31 2.930.410.39 query32 3.230.490.48 query33 2.972.932.94 query34 15.44 4.264.26 query35 4.284.274.28 query36 0.680.490.49 query37 0.190.170.16 query38 0.160.160.15 query39 0.040.030.04 query40 0.150.130.14 query41 0.100.050.04 query42 0.050.040.04 query43 0.040.050.03 Total cold run time: 107.26 s Total hot run time: 30.79 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](iceberg)]support doris's char/varchar to iceberg's string [doris]
wuwenchi commented on PR #38807: URL: https://github.com/apache/doris/pull/38807#issuecomment-2266685331 run external -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](iceberg)]support doris's char/varchar to iceberg's string [doris]
wuwenchi commented on PR #38807: URL: https://github.com/apache/doris/pull/38807#issuecomment-2266685459 run p1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](case) fix testcase out file upper case [doris]
xiaokang commented on PR #38815: URL: https://github.com/apache/doris/pull/38815#issuecomment-2266689528 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[I] Release Note 2.0.14 [doris]
xiaokang opened a new issue, #38819: URL: https://github.com/apache/doris/issues/38819 ### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description _No response_ ### Solution _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] Release Note 2.0.14 [doris]
xiaokang commented on issue #38819: URL: https://github.com/apache/doris/issues/38819#issuecomment-2266704378 中文版本： # 1 行为变更 NA # 2 新功能 - 增加获取最近一个查询 Profile 的 REST 接口 `curl http://user:password@127.0.0.1:8030/api/profile/text` - https://github.com/apache/doris/pull/38268 # 3 改进和优化 - 优化 MOW 表带有 sequence 列的主键点查性能 - https://github.com/apache/doris/pull/38287 - 优化倒排索引在查询条件很多时的性能 - https://github.com/apache/doris/pull/35346 - 创建带分词的倒排索引时,自动开启 `support_phrase` 选项加速 `match_phrase` 系列短语查询 - https://github.com/apache/doris/pull/37949 - 支持简化的 SQL hint, 例如 `SELECT /*+ query_timeout(3000) */ * FROM t;` - https://github.com/apache/doris/pull/37720 - 读对象存储遇到 429 错误时自动重试提升稳定性 - https://github.com/apache/doris/pull/35396 - 优化满足特定条件的 JOIN 性能 - https://github.com/apache/doris/pull/34703 - 避免非法数据返回 MySQL 结果时出发 coredump - https://github.com/apache/doris/pull/28069 - 输出类型名字时统一使用小写, 保持跟 MySQL 兼容对 BI 工具更加友好 - https://github.com/apache/doris/pull/38521 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](case) fix testcase out file upper case [doris]
xiaokang commented on PR #38815: URL: https://github.com/apache/doris/pull/38815#issuecomment-2266706017 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] support get json raw data with jsonpath [doris]
zeroguar opened a new pull request, #38820: URL: https://github.com/apache/doris/pull/38820 ## Proposed changes Issue Number: close #38638 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] support get json raw data with jsonpath [doris]
doris-robot commented on PR #38820: URL: https://github.com/apache/doris/pull/38820#issuecomment-2266716916 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] support get json raw data with jsonpath [doris]
github-actions[bot] commented on code in PR #38820:
URL: https://github.com/apache/doris/pull/38820#discussion_r1702777933
##
be/test/exprs/json_function_test.cpp:
##
@@ -126,5 +126,27 @@ TEST_F(JsonFunctionTest, json_path_test) {
EXPECT_TRUE(wrap_explicitly);
}
}
+TEST_F(JsonFunctionTest, extract_from_object_test) {
+std::string json_raw_data("{\"a\":\"a1\", \"b\":\"b1\", \"c\":\"c1\"}");
+json_raw_data.reserve(json_raw_data.size() + simdjson::SIMDJSON_PADDING);
Review Comment:
warning: escaped string literal can be written as a raw string literal
[modernize-raw-string-literal]
```suggestion
R"({"a":"a1", "b":"b1", "c":"c1"})");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] support get json raw data with jsonpath [doris]
github-actions[bot] commented on code in PR #38820:
URL: https://github.com/apache/doris/pull/38820#discussion_r1702779962
##
be/test/exprs/json_function_test.cpp:
##
@@ -126,5 +126,27 @@ TEST_F(JsonFunctionTest, json_path_test) {
EXPECT_TRUE(wrap_explicitly);
}
}
+TEST_F(JsonFunctionTest, extract_from_object_test) {
+std::string json_raw_data("{\"a\":\"a1\", \"b\":\"b1\", \"c\":\"c1\"}");
Review Comment:
warning: escaped string literal can be written as a raw string literal
[modernize-raw-string-literal]
```suggestion
std::string json_raw_data(R"({"a":"a1", "b":"b1", "c":"c1"})");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Add iceberg build docs. [doris-website]
morningman merged PR #940: URL: https://github.com/apache/doris-website/pull/940 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch master updated: Add iceberg build docs. (#940)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 2aa4f8307ba Add iceberg build docs. (#940)
2aa4f8307ba is described below
commit 2aa4f8307bab190b53a8fb17b0b92e154cfd123e
Author: Qi Chen
AuthorDate: Sat Aug 3 22:19:01 2024 +0800
Add iceberg build docs. (#940)
Co-authored-by: morningman
---
docs/admin-manual/resource-admin/compute-node.md | 2 +-
docs/lakehouse/datalake-building/iceberg-build.md | 297
.../admin-manual/resource-admin/compute-node.md| 4 +-
.../lakehouse/datalake-building/iceberg-build.md | 298 +
.../admin-manual/resource-admin/compute-node.md| 4 +-
.../lakehouse/datalake-building/iceberg-build.md | 298 +
.../admin-manual/resource-admin/compute-node.md| 4 +-
.../lakehouse/datalake-building/iceberg-build.md | 298 +
sidebars.json | 3 +-
.../admin-manual/resource-admin/compute-node.md| 2 +-
.../lakehouse/datalake-building/iceberg-build.md | 297
.../admin-manual/resource-admin/compute-node.md| 2 +-
.../lakehouse/datalake-building/iceberg-build.md | 297
versioned_sidebars/version-2.1-sidebars.json | 3 +-
versioned_sidebars/version-3.0-sidebars.json | 3 +-
15 files changed, 1800 insertions(+), 12 deletions(-)
diff --git a/docs/admin-manual/resource-admin/compute-node.md
b/docs/admin-manual/resource-admin/compute-node.md
index 9438f3c0ef6..b40efe2227a 100644
--- a/docs/admin-manual/resource-admin/compute-node.md
+++ b/docs/admin-manual/resource-admin/compute-node.md
@@ -94,7 +94,7 @@ At the same time, the following FE configuration will affect
the usage strategy
>
> 4. If the `min_backend_num_for_external_table` value is greater than the
> total number of BE nodes, at most number of BE will be selected.
>
-> 5. The above configurations can be modified at runtime without restarting
the FE node. And all FE nodes need to be configured.
+> 5. The above parameters can be modified dynamically through the `ADMIN SET
FRONTEND CONFIG` command without restarting the FE node. And all FE nodes need
to be configured. Or add configuration in `fe.conf` and restart the FE node.
## Best Practices
diff --git a/docs/lakehouse/datalake-building/iceberg-build.md
b/docs/lakehouse/datalake-building/iceberg-build.md
new file mode 100644
index 000..bdba7ff0ee2
--- /dev/null
+++ b/docs/lakehouse/datalake-building/iceberg-build.md
@@ -0,0 +1,297 @@
+---
+{
+"title": "Iceberg",
+"language": "en"
+}
+---
+
+
+
+Since version 2.1.6, Apache Doris supports DDL and DML operations on Iceberg.
Users can directly create library tables in Iceberg through Apache Doris and
write data to Iceberg tables. With this feature, users can perform full data
querying and writing operations on Iceberg through Apache Doris, further
simplifying the lakehouse architecture for users.
+
+This article introduces the Iceberg operations, syntax, and usage notes
supported in Apache Doris.
+
+:::tip
+This is an experimental feature.
+:::
+
+:::tip
+Before using, please set:
+
+set global enable_nereids_planner = true;
+set global enable_fallback_to_original_planner = false;
+
+For clusters upgraded from older versions, these parameters may have changed.
+:::
+
+## Metadata Creation and Deletion
+
+### Catalog
+
+- Creation
+
+```
+CREATE CATALOG [IF NOT EXISTS] iceberg PROPERTIES (
+"type" = "iceberg",
+"iceberg.catalog.type" = "hms",
+"hive.metastore.uris" = "thrift://172.21.16.47:7004",
+"warehouse" = "hdfs://172.21.16.47:4007/user/hive/warehouse/",
+"hadoop.username" = "hadoop",
+"fs.defaultFS" = "hdfs://172.21.16.47:4007"
+);
+```
+
+The above mainly demonstrates how to create an HMS Iceberg Catalog in
Apache Doris. Apache Doris currently supports multiple types of Iceberg
Catalogs. For more configurations, please refer to [Iceberg
Catalog](../datalake-analytics/iceberg.md).
+
+Note that if you need to create Iceberg tables or write data through HMS
Catalog in Apache Doris, you need to explicitly add the `fs.defaultFS` property
and `warehouse` property in the Catalog attributes. If the Catalog is created
only for querying, these two parameters can be omitted.
+
+- Deletion
+
+```
+DROP CATALOG [IF EXISTS] iceberg;
+```
+
+Deleting the Catalog does not remove any library table information in
Iceberg. It simply removes the mapping of this Iceberg Catalog in Apache Doris.
+
+### Database
+
+- Creation
+
+You can switch to the corresponding Catalog and execute the `CREATE
DATABASE` statement:
+
+```
+SWITCH iceberg;
+CREATE DATABASE
Re: [PR] [fix](case) fix testcase out file upper case [doris]
xiaokang merged PR #38815: URL: https://github.com/apache/doris/pull/38815 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](case) fix testcase out file upper case (#38815)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.0 by this push: new f791b95b3be [fix](case) fix testcase out file upper case (#38815) f791b95b3be is described below commit f791b95b3be6a590c3f4dcbf053527eb56ed2463 Author: Kang AuthorDate: Sat Aug 3 22:20:54 2024 +0800 [fix](case) fix testcase out file upper case (#38815) related #38012 --- .../schema_change_modify_mv_column_type.out| 148 ++--- .../time_type/test_time_in_runtimepredicate.groovy | 2 +- .../test_array_with_large_dataset.groovy | 6 +- .../function/test_not_supported_agg_fn.groovy | 2 +- 4 files changed, 77 insertions(+), 81 deletions(-) diff --git a/regression-test/data/schema_change_p0/schema_change_modify_mv_column_type.out b/regression-test/data/schema_change_p0/schema_change_modify_mv_column_type.out index 4e6f3c7a5eb..f9980ff895b 100644 --- a/regression-test/data/schema_change_p0/schema_change_modify_mv_column_type.out +++ b/regression-test/data/schema_change_p0/schema_change_modify_mv_column_type.out @@ -1,42 +1,42 @@ -- This file is automatically generated. You should know what you did if you want to edit this -- !sql -- -tbl_scalar_types_dup DUP_KEYSk1 BIGINT BIGINT Yes true \N true - c_bool BOOLEAN BOOLEAN Yes false \N NONEtrue - c_tinyint TINYINT TINYINT Yes false \N NONE true - c_smallint SMALLINTSMALLINTYes false \N NONEtrue - c_int INT INT Yes false \N NONEtrue - c_bigintBIGINT BIGINT Yes false \N NONE true - c_largeint LARGEINTLARGEINTYes false \N NONEtrue - c_float FLOAT FLOAT Yes false \N NONEtrue - c_doubleDOUBLE DOUBLE Yes false \N NONE true - c_decimal DECIMAL(20, 3) DECIMALV3(20, 3)Yes false \N NONEtrue - c_decimalv3 DECIMAL(20, 3) DECIMALV3(20, 3)Yes false \N NONEtrue - c_date DATEDATEV2 Yes false \N NONEtrue - c_datetime DATETIMEDATETIMEV2(0) Yes false \N NONEtrue - c_datev2DATEDATEV2 Yes false \N NONE true - c_datetimev2DATETIMEDATETIMEV2(0) Yes false \N NONEtrue - c_char CHAR(15)CHAR(15)Yes false \N NONEtrue - c_varchar VARCHAR(100)VARCHAR(100)Yes false \N NONEtrue - c_stringTEXTTEXTYes false \N NONE true - -mv_tbl_scalar_types_dup_1 DUP_KEYSmv_c_tinyintTINYINT TINYINT Yes true\N true`c_tinyint` - mv_c_bool BOOLEAN BOOLEAN Yes true\N true`c_bool` - mv_k1 BIGINT BIGINT Yes true\N true `k1` - mv_c_smallint SMALLINTSMALLINTYes false \N NONEtrue`c_smallint` - mv_c_intINT INT Yes false \N NONE true`c_int` - mv_c_bigint BIGINT BIGINT Yes false \N NONE true`c_bigint` - mv_c_largeint LARGEINTLARGEINTYes false \N NONEtrue`c_largeint` - mv_c_float FLOAT FLOAT Yes false \N NONE true`c_float` - mv_c_double DOUBLE DOUBLE Yes false \N NONE true`c_double` - mv_c_decimalDECIMAL(20, 3) DECIMALV3(20, 3)Yes false \N NONEtrue`c_decimal` - mv_c_decimalv3 DECIMAL(20, 3) DECIMALV3(20, 3)Yes false \N NONEtrue`c_decimalv3` - mv_c_date DATEDATEV2 Yes false \N NONE true`c_date` - mv_c_datetime DATETIMEDATETIMEV2(0) Yes false \N NONEtrue`c_datetime` - mv_c_datev2 DATEDATEV2 Yes false \N NONE true`c_datev2` - mv_c_datetimev2 DATETIMEDATETIMEV2(0) Yes
Re: [PR] support get json raw data with jsonpath [doris]
github-actions[bot] commented on PR #38820: URL: https://github.com/apache/doris/pull/38820#issuecomment-2266733047 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](test)Fix unstable test drop index fault #38768 [doris]
yiguolei merged PR #38772: URL: https://github.com/apache/doris/pull/38772 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](test)Fix unstable test drop index fault #38768 (#38772)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 74908c123a5 [fix](test)Fix unstable test drop index fault #38768
(#38772)
74908c123a5 is described below
commit 74908c123a53d1bf946177c515244cdb01b9bbba
Author: qiye
AuthorDate: Sat Aug 3 23:47:55 2024 +0800
[fix](test)Fix unstable test drop index fault #38768 (#38772)
## Proposed changes
bp #38768
---
...index_builder_drop_index_fault_injection.groovy | 22 ++
1 file changed, 22 insertions(+)
diff --git
a/regression-test/suites/fault_injection_p0/test_index_builder_drop_index_fault_injection.groovy
b/regression-test/suites/fault_injection_p0/test_index_builder_drop_index_fault_injection.groovy
index f8501f0e851..353afdf0769 100644
---
a/regression-test/suites/fault_injection_p0/test_index_builder_drop_index_fault_injection.groovy
+++
b/regression-test/suites/fault_injection_p0/test_index_builder_drop_index_fault_injection.groovy
@@ -18,6 +18,24 @@
import org.codehaus.groovy.runtime.IOGroovyMethods
suite("test_index_builder_drop_index_fault_injection", "nonConcurrent") {
+def timeout = 6
+def delta_time = 1000
+def alter_res = "null"
+def useTime = 0
+def wait_for_latest_op_on_table_finish = { table_name, OpTimeout ->
+for(int t = delta_time; t <= OpTimeout; t += delta_time){
+alter_res = sql """SHOW ALTER TABLE COLUMN WHERE TableName =
"${table_name}" ORDER BY CreateTime DESC LIMIT 1;"""
+alter_res = alter_res.toString()
+if(alter_res.contains("FINISHED")) {
+sleep(3000) // wait change table state to normal
+logger.info(table_name + " latest alter job finished, detail:
" + alter_res)
+break
+}
+useTime = t
+sleep(delta_time)
+}
+assertTrue(useTime <= OpTimeout, "wait_for_latest_op_on_table_finish
timeout")
+}
def runTest = { indexTbName ->
sql """ insert into ${indexTbName} values(1, "json love anny", "json",
"anny",1); """
sql "sync"
@@ -33,6 +51,7 @@ suite("test_index_builder_drop_index_fault_injection",
"nonConcurrent") {
try {
GetDebugPoint().enableDebugPointForAllBEs("index_builder.update_inverted_index_info.drop_index",
[indexes_count: 3])
sql "DROP INDEX index_int ON ${indexTbName}"
+wait_for_latest_op_on_table_finish(indexTbName, timeout)
show_result = sql_return_maparray "show index from ${indexTbName}"
logger.info("show index from " + indexTbName + " result: " +
show_result)
assertEquals(show_result.size(), 3)
@@ -46,6 +65,7 @@ suite("test_index_builder_drop_index_fault_injection",
"nonConcurrent") {
try {
GetDebugPoint().enableDebugPointForAllBEs("index_builder.update_inverted_index_info.drop_index",
[indexes_count: 2])
sql "DROP INDEX index_str_k2 ON ${indexTbName}"
+wait_for_latest_op_on_table_finish(indexTbName, timeout)
show_result = sql_return_maparray "show index from ${indexTbName}"
logger.info("show index from " + indexTbName + " result: " +
show_result)
assertEquals(show_result.size(), 2)
@@ -58,6 +78,7 @@ suite("test_index_builder_drop_index_fault_injection",
"nonConcurrent") {
try {
GetDebugPoint().enableDebugPointForAllBEs("index_builder.update_inverted_index_info.drop_index",
[indexes_count: 1])
sql "DROP INDEX index_str_k4 ON ${indexTbName}"
+wait_for_latest_op_on_table_finish(indexTbName, timeout)
show_result = sql_return_maparray "show index from ${indexTbName}"
logger.info("show index from " + indexTbName + " result: " +
show_result)
assertEquals(show_result.size(), 1)
@@ -69,6 +90,7 @@ suite("test_index_builder_drop_index_fault_injection",
"nonConcurrent") {
try {
GetDebugPoint().enableDebugPointForAllBEs("index_builder.update_inverted_index_info.drop_index",
[indexes_count: 0])
sql "DROP INDEX index_k5 ON ${indexTbName}"
+wait_for_latest_op_on_table_finish(indexTbName, timeout)
show_result = sql_return_maparray "show index from ${indexTbName}"
logger.info("show index from " + indexTbName + " result: " +
show_result)
assertEquals(show_result.size(), 0)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](test)Fix build index fault test (#38736) (#38762)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new fe3e3d0fab6 [fix](test)Fix build index fault test (#38736) (#38762)
fe3e3d0fab6 is described below
commit fe3e3d0fab68ef9baea50c845b265e23924400d2
Author: qiye
AuthorDate: Sat Aug 3 23:48:29 2024 +0800
[fix](test)Fix build index fault test (#38736) (#38762)
## Proposed changes
backport #38736
---
.../fault_injection_p0/test_build_index_fault.groovy | 14 --
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git
a/regression-test/suites/fault_injection_p0/test_build_index_fault.groovy
b/regression-test/suites/fault_injection_p0/test_build_index_fault.groovy
index 368314cc1af..056295f98bc 100644
--- a/regression-test/suites/fault_injection_p0/test_build_index_fault.groovy
+++ b/regression-test/suites/fault_injection_p0/test_build_index_fault.groovy
@@ -216,14 +216,15 @@ suite("test_build_index_fault", "inverted_index,
nonConcurrent,p2"){
GetDebugPoint().enableDebugPointForAllBEs("fault_inject::BetaRowset::link_files_to::_link_inverted_index_file")
sql """ BUILD INDEX idx_title ON ${tableName}; """
state = wait_for_last_build_index_on_table_finish(tableName, timeout)
-assertEquals("wait_timeout", state)
+assertEquals("CANCELLED", state)
// check data
qt_count5 """ SELECT COUNT() from ${tableName}; """
// disable error_inject for BetaRowset link inverted index file and expect
state is FINISHED
GetDebugPoint().disableDebugPointForAllBEs("fault_inject::BetaRowset::link_files_to::_link_inverted_index_file")
-// timeout * 10 for possible fe schedule delay
-state = wait_for_last_build_index_on_table_finish(tableName, timeout * 10)
+// rebuild index
+sql """ BUILD INDEX idx_title ON ${tableName}; """
+state = wait_for_last_build_index_on_table_finish(tableName, timeout)
assertEquals("FINISHED", state)
// check data
qt_count6 """ SELECT COUNT() from ${tableName}; """
@@ -233,13 +234,14 @@ suite("test_build_index_fault", "inverted_index,
nonConcurrent,p2"){
GetDebugPoint().enableDebugPointForAllBEs("IndexBuilder::handle_single_rowset")
sql """ BUILD INDEX idx_url ON ${tableName}; """
state = wait_for_last_build_index_on_table_finish(tableName, timeout)
-assertEquals("wait_timeout", state)
+assertEquals("CANCELLED", state)
// check data
qt_count7 """ SELECT COUNT() from ${tableName}; """
GetDebugPoint().disableDebugPointForAllBEs("IndexBuilder::handle_single_rowset")
-// timeout * 10 for possible fe schedule delay
-state = wait_for_last_build_index_on_table_finish(tableName, timeout * 10)
+// rebuild index
+sql """ BUILD INDEX idx_url ON ${tableName}; """
+state = wait_for_last_build_index_on_table_finish(tableName, timeout)
assertEquals("FINISHED", state)
// check data
qt_count8 """ SELECT COUNT() from ${tableName}; """
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ES Catalog)Fix unstable test test_es_query (#38801) [doris]
yiguolei merged PR #38802: URL: https://github.com/apache/doris/pull/38802 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](test)Fix build index fault test (#38736) [doris]
yiguolei merged PR #38762: URL: https://github.com/apache/doris/pull/38762 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](ES Catalog)Fix unstable test test_es_query (#38801) (#38802)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new c0caca7c557 [fix](ES Catalog)Fix unstable test test_es_query (#38801)
(#38802)
c0caca7c557 is described below
commit c0caca7c55725ea4808e8ae8240e74694ad5671f
Author: qiye
AuthorDate: Sat Aug 3 23:49:00 2024 +0800
[fix](ES Catalog)Fix unstable test test_es_query (#38801) (#38802)
## Proposed changes
bp #38801
---
docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh | 1 +
1 file changed, 1 insertion(+)
diff --git
a/docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh
b/docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh
index 21f1dc55a33..575118d8db9 100755
--- a/docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh
+++ b/docker/thirdparties/docker-compose/elasticsearch/scripts/es_init.sh
@@ -137,3 +137,4 @@ curl "http://${ES_8_HOST}:9200/test3_20231005/_doc/1"; -H
"Content-Type:applicati
# put _meta for array
curl "http://${ES_8_HOST}:9200/test1/_mapping"; -H
"Content-Type:application/json" -X PUT -d "@/mnt/scripts/index/array_meta.json"
curl "http://${ES_8_HOST}:9200/test2_20220808/_mapping"; -H
"Content-Type:application/json" -X PUT -d "@/mnt/scripts/index/array_meta.json"
+curl "http://${ES_8_HOST}:9200/test2_20220809/_mapping"; -H
"Content-Type:application/json" -X PUT -d "@/mnt/scripts/index/array_meta.json"
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](info) processlist schema table support show all fe [doris]
yiguolei commented on PR #38701: URL: https://github.com/apache/doris/pull/38701#issuecomment-2266870684 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](info) processlist schema table support show all fe [doris]
github-actions[bot] commented on PR #38701: URL: https://github.com/apache/doris/pull/38701#issuecomment-2266887643 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](info) processlist schema table support show all fe [doris]
doris-robot commented on PR #38701: URL: https://github.com/apache/doris/pull/38701#issuecomment-2267081797 TPC-H: Total hot run time: 41760 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit ffe52166383c0df2645e33d88a8c3a1918e048d5, data reload: false -- Round 1 -- q1 17633 422140924092 q2 2043197 195 195 q3 10533 132113491321 q4 10191 907 1029907 q5 7729300729762976 q6 219 139 137 137 q7 1050615 607 607 q8 9439185719671857 q9 8446661166026602 q10 8724382638333826 q11 427 248 246 246 q12 416 225 227 225 q13 17749 293329382933 q14 268 251 242 242 q15 531 497 488 488 q16 530 398 387 387 q17 986 919 919 919 q18 8159729372877287 q19 1450122612291226 q20 583 319 339 319 q21 5260469148394691 q22 351 277 286 277 Total cold run time: 112717 ms Total hot run time: 41760 ms - Round 2, with runtime_filter_mode=off - q1 4173405440124012 q2 328 233 223 223 q3 2999302331883023 q4 2038203019671967 q5 5576549054575457 q6 219 133 129 129 q7 2138182417961796 q8 3344337233723372 q9 8668871487108710 q10 3945403939293929 q11 565 457 466 457 q12 787 601 582 582 q13 16345 311131363111 q14 302 264 291 264 q15 519 493 478 478 q16 469 419 426 419 q17 1780172517311725 q18 8330780377827782 q19 1717174817221722 q20 2032182418331824 q21 5694533454015334 q22 545 449 465 449 Total cold run time: 72513 ms Total hot run time: 56765 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](info) processlist schema table support show all fe [doris]
doris-robot commented on PR #38701: URL: https://github.com/apache/doris/pull/38701#issuecomment-2267084341 TPC-DS: Total hot run time: 169826 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit ffe52166383c0df2645e33d88a8c3a1918e048d5, data reload: false query1 910 375 364 364 query2 6477169116601660 query3 6648210 229 210 query4 20420 17738 17317 17317 query5 3613506 525 506 query6 281 184 160 160 query7 4590306 299 299 query8 256 201 203 201 query9 8521231823052305 query10 426 291 280 280 query11 10684 10093 10128 10093 query12 118 87 88 87 query13 1629386 382 382 query14 9222758974307430 query15 210 161 162 161 query16 6974409 440 409 query17 931 546 546 546 query18 1915283 278 278 query19 190 141 139 139 query20 88 85 84 84 query21 223 98 95 95 query22 4301405539483948 query23 33732 33638 33707 33638 query24 8647309930703070 query25 653 434 402 402 query26 729 155 156 155 query27 2392293 286 286 query28 5320199619641964 query29 913 428 433 428 query30 242 152 167 152 query31 960 786 768 768 query32 105 52 61 52 query33 595 326 330 326 query34 958 496 510 496 query35 872 781 763 763 query36 1077907 873 873 query37 148 92 84 84 query38 2937278927982789 query39 903 798 798 798 query40 200 116 114 114 query41 47 49 79 49 query42 118 111 101 101 query43 454 419 421 419 query44 1137722 727 722 query45 225 179 174 174 query46 1077828 766 766 query47 1832171716961696 query48 377 293 295 293 query49 841 412 434 412 query50 892 432 430 430 query51 6828659766296597 query52 95 95 89 89 query53 245 178 175 175 query54 617 453 448 448 query55 78 75 72 72 query56 291 250 248 248 query57 1119104310451043 query58 258 267 292 267 query59 2716248623332333 query60 298 274 265 265 query61 95 97 91 91 query62 886 664 651 651 query63 248 175 177 175 query64 3987191218951895 query65 3176308131153081 query66 825 325 322 322 query67 15254 14780 15024 14780 query68 4341569 568 568 query69 451 297 303 297 query70 1099102710651027 query71 374 274 272 272 query72 7078267424932493 query73 763 332 326 326 query74 6122568856695669 query75 3375269226952692 query76 2109122112871221 query77 471 307 300 300 query78 9407896588548854 query79 1230545 521 521 query80 1675524 515 515 query81 568 227 226 226 query82 1127130 130 130 query83 280 171 171 171 query84 275 79 78 78 query85 1092301 286 286 query86 392 312 273 273 query87 3289311931443119 query88 2989248824772477 query89 387 297 290 290 query90 1706198 189 189 query91 123 102 98 98 query92 58 49 49 49 query93 1293622 616 616 query94 765 293 281 281 query95 382 272 336 272 query96 599 288 286 286 query97 3270306930423042 query98 218 205 199 199 query99 1717128712831283 Total cold run time: 253051 ms Total hot run time: 169826 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [opt](info) processlist schema table support show all fe [doris]
doris-robot commented on PR #38701: URL: https://github.com/apache/doris/pull/38701#issuecomment-2267085596 ClickBench: Total hot run time: 29.83 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit ffe52166383c0df2645e33d88a8c3a1918e048d5, data reload: false query1 0.040.040.04 query2 0.070.040.04 query3 0.220.060.05 query4 1.670.080.07 query5 0.490.500.49 query6 1.140.720.71 query7 0.020.020.01 query8 0.050.040.04 query9 0.580.520.52 query10 0.590.560.56 query11 0.160.120.12 query12 0.150.120.12 query13 0.620.590.60 query14 0.770.780.77 query15 0.890.860.87 query16 0.340.350.35 query17 1.010.991.01 query18 0.220.210.20 query19 1.861.751.70 query20 0.010.010.01 query21 15.40 0.790.67 query22 4.157.701.21 query23 17.82 1.281.28 query24 2.260.230.22 query25 0.180.080.07 query26 0.310.210.21 query27 0.460.230.23 query28 13.16 0.990.96 query29 12.54 3.263.25 query30 0.250.060.05 query31 2.880.400.40 query32 3.250.500.48 query33 2.972.932.94 query34 15.44 4.284.28 query35 4.294.324.27 query36 0.680.480.49 query37 0.200.170.16 query38 0.160.150.15 query39 0.040.030.04 query40 0.150.130.14 query41 0.100.040.05 query42 0.060.050.05 query43 0.050.040.04 Total cold run time: 107.7 s Total hot run time: 29.83 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] (doris-future) Support auto partition name function [doris]
wyxxxcat commented on PR #34258: URL: https://github.com/apache/doris/pull/34258#issuecomment-2267207028 run cloud p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](scan) support parallel scan for aggregate key and unique key with mor [doris]
github-actions[bot] closed pull request #30729: [feat](scan) support parallel scan for aggregate key and unique key with mor URL: https://github.com/apache/doris/pull/30729 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix][optimize] optimize the coding import Lists dependency package [doris]
github-actions[bot] closed pull request #30176: [fix][optimize] optimize the coding import Lists dependency package URL: https://github.com/apache/doris/pull/30176 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](repository) support alter s3 repo info about ak/sk/token [doris]
github-actions[bot] closed pull request #30147: [enhancement](repository) support alter s3 repo info about ak/sk/token URL: https://github.com/apache/doris/pull/30147 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [release](version) change branch-2.0 version to 2.0.14 [doris]
xiaokang merged PR #38735: URL: https://github.com/apache/doris/pull/38735 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [release](version) change branch-2.0 version to 2.0.14 (#38735)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new eaaf7260191 [release](version) change branch-2.0 version to 2.0.14
(#38735)
eaaf7260191 is described below
commit eaaf7260191b36b0f6fa77afdc74bc0a2badb9bd
Author: Kang
AuthorDate: Sun Aug 4 09:22:52 2024 +0800
[release](version) change branch-2.0 version to 2.0.14 (#38735)
---
gensrc/script/gen_build_version.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/gensrc/script/gen_build_version.sh
b/gensrc/script/gen_build_version.sh
index 5161addbc68..b4be5922f90 100755
--- a/gensrc/script/gen_build_version.sh
+++ b/gensrc/script/gen_build_version.sh
@@ -30,7 +30,7 @@ set -eo pipefail
build_version_prefix="doris"
build_version_major=2
build_version_minor=0
-build_version_patch=13
+build_version_patch=14
build_version_rc_version=""
if [[ -z "${build_version_rc_version}" ]]; then
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) tag 2.0.14 created (now eaaf7260191)
This is an automated email from the ASF dual-hosted git repository. kxiao pushed a change to tag 2.0.14 in repository https://gitbox.apache.org/repos/asf/doris.git at eaaf7260191 (commit) No new revisions were added by this update. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
svn commit: r70681 - in /dev/doris/2.0.14: ./ apache-doris-2.0.14-src.tar.gz apache-doris-2.0.14-src.tar.gz.asc apache-doris-2.0.14-src.tar.gz.sha512
Author: kxiao Date: Sun Aug 4 01:34:34 2024 New Revision: 70681 Log: doris-2.0.14 release vote Added: dev/doris/2.0.14/ dev/doris/2.0.14/apache-doris-2.0.14-src.tar.gz (with props) dev/doris/2.0.14/apache-doris-2.0.14-src.tar.gz.asc (with props) dev/doris/2.0.14/apache-doris-2.0.14-src.tar.gz.sha512 Added: dev/doris/2.0.14/apache-doris-2.0.14-src.tar.gz == Binary file - no diff available. Propchange: dev/doris/2.0.14/apache-doris-2.0.14-src.tar.gz -- svn:mime-type = application/x-gzip Added: dev/doris/2.0.14/apache-doris-2.0.14-src.tar.gz.asc == Binary file - no diff available. Propchange: dev/doris/2.0.14/apache-doris-2.0.14-src.tar.gz.asc -- svn:mime-type = application/pgp-signature Added: dev/doris/2.0.14/apache-doris-2.0.14-src.tar.gz.sha512 == --- dev/doris/2.0.14/apache-doris-2.0.14-src.tar.gz.sha512 (added) +++ dev/doris/2.0.14/apache-doris-2.0.14-src.tar.gz.sha512 Sun Aug 4 01:34:34 2024 @@ -0,0 +1 @@ +336a848556736342213664590bdeb31bc55a11a1d7ff31438f8ca2a1cdc17454f7e1ceccb6ae9b0009bb3f31afa33fc029786d5fe6cf51892002dbb8dc865b47 apache-doris-2.0.14-src.tar.gz - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [docs](function) support auto_partition_name function [doris-website]
wyxxxcat opened a new pull request, #945: URL: https://github.com/apache/doris-website/pull/945 https://github.com/apache/doris/pull/34258 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: [branch-3.0](log) add tablet clear cache log (#38715)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 9daf67e3384 [branch-3.0](log) add tablet clear cache log (#38715)
9daf67e3384 is described below
commit 9daf67e3384acb57e9dc606fce95f3e2df8465f7
Author: yujun
AuthorDate: Sun Aug 4 10:08:29 2024 +0800
[branch-3.0](log) add tablet clear cache log (#38715)
---
be/src/agent/task_worker_pool.cpp | 4 +++-
be/src/olap/tablet.cpp| 1 +
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/be/src/agent/task_worker_pool.cpp
b/be/src/agent/task_worker_pool.cpp
index efd15d0711b..bfa5b888edf 100644
--- a/be/src/agent/task_worker_pool.cpp
+++ b/be/src/agent/task_worker_pool.cpp
@@ -1518,7 +1518,9 @@ void create_tablet_callback(StorageEngine& engine, const
TAgentTaskRequest& req)
COUNTER_UPDATE(profile->total_time_counter(), elapsed_time);
std::stringstream ss;
profile->pretty_print(&ss);
-LOG(WARNING) << "create tablet cost(s) " << elapsed_time / 1e9 <<
std::endl << ss.str();
+LOG(WARNING) << "create tablet " << create_tablet_req.tablet_id <<
" cost(s) "
+ << elapsed_time / 1e9 << std::endl
+ << ss.str();
}
});
DorisMetrics::instance()->create_tablet_requests_total->increment(1);
diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp
index 1a1d3be6bc9..5018c70be11 100644
--- a/be/src/olap/tablet.cpp
+++ b/be/src/olap/tablet.cpp
@@ -2642,6 +2642,7 @@ Status Tablet::ingest_binlog_metas(RowsetBinlogMetasPB*
metas_pb) {
void Tablet::clear_cache() {
std::shared_lock rlock(get_header_lock());
+SCOPED_SIMPLE_TRACE_IF_TIMEOUT(TRACE_TABLET_LOCK_THRESHOLD);
static auto recycle_segment_cache = [](const auto& rowset_map) {
for (auto& [_, rowset] : rowset_map) {
rowset->clear_cache();
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-3.0](log) add tablet clear cache log [doris]
dataroaring merged PR #38715: URL: https://github.com/apache/doris/pull/38715 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](schema-change) Fix schema change run into nullable check specified for MV [doris]
github-actions[bot] commented on PR #38806: URL: https://github.com/apache/doris/pull/38806#issuecomment-2267250147 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](schema-change) Fix schema change run into nullable check specified for MV [doris]
github-actions[bot] commented on PR #38806: URL: https://github.com/apache/doris/pull/38806#issuecomment-2267250172 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](group commit) Fix multiple cluster group commit BE select strategy [doris]
github-actions[bot] commented on PR #38644: URL: https://github.com/apache/doris/pull/38644#issuecomment-2267251014 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](mow) Make calculate delete bitmap timeout configurable [doris]
dataroaring merged PR #38668: URL: https://github.com/apache/doris/pull/38668 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Improvement](mow) Make calculate delete bitmap timeout configurable (#38668)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 2da8f8bae31 [Improvement](mow) Make calculate delete bitmap timeout
configurable (#38668)
2da8f8bae31 is described below
commit 2da8f8bae31e76e6f887724708fb5b2f51475e65
Author: huanghaibin <284824...@qq.com>
AuthorDate: Sun Aug 4 10:15:34 2024 +0800
[Improvement](mow) Make calculate delete bitmap timeout configurable
(#38668)
Now the timeout of calculating delete bitmap is hard code, make it
configurable.
---
fe/fe-common/src/main/java/org/apache/doris/common/Config.java | 9 +
.../doris/cloud/transaction/CloudGlobalTransactionMgr.java | 6 ++
.../main/java/org/apache/doris/service/FrontendServiceImpl.java | 6 --
3 files changed, 15 insertions(+), 6 deletions(-)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index e67fd0a525c..3c8d0e0592a 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -2972,6 +2972,15 @@ public class Config extends ConfigBase {
@ConfField(description = {"存算分离模式下建表是否检查残留recycler key, 默认true",
"create table in cloud mode, check recycler key remained, default
true"})
public static boolean check_create_table_recycle_key_remained = true;
+
+@ConfField(mutable = true, description = {"存算分离模式下fe向ms请求锁的过期时间，默认10s"})
+public static int delete_bitmap_lock_expiration_seconds = 10;
+
+@ConfField(mutable = true, description = {"存算分离模式下calculate delete bitmap
task 超时时间，默认15s"})
+public static int calculate_delete_bitmap_task_timeout_seconds = 15;
+
+@ConfField(mutable = true, description = {"存算分离模式下commit阶段等锁超时时间，默认5s"})
+public static int try_commit_lock_timeout_seconds = 5;
// ATTN: DONOT add any config not related to cloud mode here
// ATTN: DONOT add any config not related to cloud mode here
// ATTN: DONOT add any config not related to cloud mode here
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java
index b153fd006c4..000c05fde27 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java
@@ -147,8 +147,6 @@ import java.util.stream.Collectors;
public class CloudGlobalTransactionMgr implements GlobalTransactionMgrIface {
private static final Logger LOG =
LogManager.getLogger(CloudGlobalTransactionMgr.class);
private static final String NOT_SUPPORTED_MSG = "Not supported in cloud
mode";
-private static final int DELETE_BITMAP_LOCK_EXPIRATION_SECONDS = 10;
-private static final int CALCULATE_DELETE_BITMAP_TASK_TIMEOUT_SECONDS = 15;
private TxnStateCallbackFactory callbackFactory;
private final Map subTxnIdToTxnId = new ConcurrentHashMap<>();
@@ -743,7 +741,7 @@ public class CloudGlobalTransactionMgr implements
GlobalTransactionMgrIface {
builder.setTableId(entry.getKey())
.setLockId(transactionId)
.setInitiator(-1)
-.setExpiration(DELETE_BITMAP_LOCK_EXPIRATION_SECONDS)
+
.setExpiration(Config.delete_bitmap_lock_expiration_seconds)
.setRequireCompactionStats(true);
List tabletList = tableToTabletList.get(entry.getKey());
for (Long tabletId : tabletList) {
@@ -853,7 +851,7 @@ public class CloudGlobalTransactionMgr implements
GlobalTransactionMgrIface {
boolean ok;
try {
-ok =
countDownLatch.await(CALCULATE_DELETE_BITMAP_TASK_TIMEOUT_SECONDS,
TimeUnit.SECONDS);
+ok =
countDownLatch.await(Config.calculate_delete_bitmap_task_timeout_seconds,
TimeUnit.SECONDS);
} catch (InterruptedException e) {
LOG.warn("InterruptedException: ", e);
ok = false;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
index 47bcff7c6f5..56d2cb01e6a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
@@ -1679,7 +1679,8 @@ public class FrontendServiceImpl implements
FrontendService.Iface {
+ request.isSetDbId() + " id: " +
Long.toString(request.isSetDbId() ? request.getDbId() : 0)
+ " fullDbName: " + fullDbName);
}
-long timeoutMs = request.isSetThriftR
Re: [PR] [Enhancement](merge-on-write) Support dynamic delete bitmap txn_delete_bitmap_cache [doris]
github-actions[bot] commented on PR #38761: URL: https://github.com/apache/doris/pull/38761#issuecomment-2267252453 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](merge-on-write) Support dynamic delete bitmap txn_delete_bitmap_cache [doris]
github-actions[bot] commented on PR #38761: URL: https://github.com/apache/doris/pull/38761#issuecomment-2267252464 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated (9daf67e3384 -> afe4ab72dae)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from 9daf67e3384 [branch-3.0](log) add tablet clear cache log (#38715) new 15906e3bf88 [Improvement](mow) Make calculate delete bitmap timeout configurable (#38668) new afe4ab72dae [Enhancement](merge-on-write) Support dynamic delete bitmap txn_delete_bitmap_cache (#38761) The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/cloud/cloud_storage_engine.cpp| 12 ++-- .../src/main/java/org/apache/doris/common/Config.java| 9 + .../doris/cloud/transaction/CloudGlobalTransactionMgr.java | 6 ++ .../java/org/apache/doris/service/FrontendServiceImpl.java | 6 -- 4 files changed, 25 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](merge-on-write) Support dynamic delete bitmap txn_delete_bitmap_cache [doris]
dataroaring merged PR #38761: URL: https://github.com/apache/doris/pull/38761 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (2da8f8bae31 -> 6619b64606f)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 2da8f8bae31 [Improvement](mow) Make calculate delete bitmap timeout configurable (#38668) add 6619b64606f [Enhancement](merge-on-write) Support dynamic delete bitmap txn_delete_bitmap_cache (#38761) No new revisions were added by this update. Summary of changes: be/src/cloud/cloud_storage_engine.cpp | 12 ++-- 1 file changed, 10 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/02: [Improvement](mow) Make calculate delete bitmap timeout configurable (#38668)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 15906e3bf88aa370b8d73ba8190e164fcce548a6
Author: huanghaibin <284824...@qq.com>
AuthorDate: Sun Aug 4 10:15:34 2024 +0800
[Improvement](mow) Make calculate delete bitmap timeout configurable
(#38668)
Now the timeout of calculating delete bitmap is hard code, make it
configurable.
---
fe/fe-common/src/main/java/org/apache/doris/common/Config.java | 9 +
.../doris/cloud/transaction/CloudGlobalTransactionMgr.java | 6 ++
.../main/java/org/apache/doris/service/FrontendServiceImpl.java | 6 --
3 files changed, 15 insertions(+), 6 deletions(-)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 55ef9961e50..3a4467d4cb3 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -2929,6 +2929,15 @@ public class Config extends ConfigBase {
@ConfField(description = {"存算分离模式下建表是否检查残留recycler key, 默认true",
"create table in cloud mode, check recycler key remained, default
true"})
public static boolean check_create_table_recycle_key_remained = true;
+
+@ConfField(mutable = true, description = {"存算分离模式下fe向ms请求锁的过期时间，默认10s"})
+public static int delete_bitmap_lock_expiration_seconds = 10;
+
+@ConfField(mutable = true, description = {"存算分离模式下calculate delete bitmap
task 超时时间，默认15s"})
+public static int calculate_delete_bitmap_task_timeout_seconds = 15;
+
+@ConfField(mutable = true, description = {"存算分离模式下commit阶段等锁超时时间，默认5s"})
+public static int try_commit_lock_timeout_seconds = 5;
// ATTN: DONOT add any config not related to cloud mode here
// ATTN: DONOT add any config not related to cloud mode here
// ATTN: DONOT add any config not related to cloud mode here
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java
index a4d0e582471..9b25e7de6c9 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java
@@ -145,8 +145,6 @@ import java.util.stream.Collectors;
public class CloudGlobalTransactionMgr implements GlobalTransactionMgrIface {
private static final Logger LOG =
LogManager.getLogger(CloudGlobalTransactionMgr.class);
private static final String NOT_SUPPORTED_MSG = "Not supported in cloud
mode";
-private static final int DELETE_BITMAP_LOCK_EXPIRATION_SECONDS = 10;
-private static final int CALCULATE_DELETE_BITMAP_TASK_TIMEOUT_SECONDS = 15;
private TxnStateCallbackFactory callbackFactory;
private final Map subTxnIdToTxnId = new ConcurrentHashMap<>();
@@ -721,7 +719,7 @@ public class CloudGlobalTransactionMgr implements
GlobalTransactionMgrIface {
builder.setTableId(entry.getKey())
.setLockId(transactionId)
.setInitiator(-1)
-.setExpiration(DELETE_BITMAP_LOCK_EXPIRATION_SECONDS)
+
.setExpiration(Config.delete_bitmap_lock_expiration_seconds)
.setRequireCompactionStats(true);
List tabletList = tableToTabletList.get(entry.getKey());
for (Long tabletId : tabletList) {
@@ -831,7 +829,7 @@ public class CloudGlobalTransactionMgr implements
GlobalTransactionMgrIface {
boolean ok;
try {
-ok =
countDownLatch.await(CALCULATE_DELETE_BITMAP_TASK_TIMEOUT_SECONDS,
TimeUnit.SECONDS);
+ok =
countDownLatch.await(Config.calculate_delete_bitmap_task_timeout_seconds,
TimeUnit.SECONDS);
} catch (InterruptedException e) {
LOG.warn("InterruptedException: ", e);
ok = false;
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
index 8bf07e81e27..58754d69fd0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java
@@ -1682,7 +1682,8 @@ public class FrontendServiceImpl implements
FrontendService.Iface {
+ request.isSetDbId() + " id: " +
Long.toString(request.isSetDbId() ? request.getDbId() : 0)
+ " fullDbName: " + fullDbName);
}
-long timeoutMs = request.isSetThriftRpcTimeoutMs() ?
request.getThriftRpcTimeoutMs() / 2 : 5000;
+long timeoutMs = request.isSetThriftRpcTimeoutMs() ?
request.getThriftRpcTimeoutMs() / 2
+: Config.try_commit_lo
(doris) 02/02: [Enhancement](merge-on-write) Support dynamic delete bitmap txn_delete_bitmap_cache (#38761)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit afe4ab72dae7e2d07fe44c1f9211f74eaa498f1b
Author: huanghaibin <284824...@qq.com>
AuthorDate: Sun Aug 4 10:16:23 2024 +0800
[Enhancement](merge-on-write) Support dynamic delete bitmap
txn_delete_bitmap_cache (#38761)
The default cache is set to 100MB, use memory limit to dynamic adjustmen
---
be/src/cloud/cloud_storage_engine.cpp | 12 ++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/be/src/cloud/cloud_storage_engine.cpp
b/be/src/cloud/cloud_storage_engine.cpp
index de4bbac7b3e..8062631af1f 100644
--- a/be/src/cloud/cloud_storage_engine.cpp
+++ b/be/src/cloud/cloud_storage_engine.cpp
@@ -48,6 +48,7 @@
#include "olap/memtable_flush_executor.h"
#include "olap/storage_policy.h"
#include "runtime/memory/cache_manager.h"
+#include "util/parse_util.h"
namespace doris {
@@ -186,8 +187,15 @@ Status CloudStorageEngine::open() {
_calc_delete_bitmap_executor =
std::make_unique();
_calc_delete_bitmap_executor->init();
-_txn_delete_bitmap_cache =
-
std::make_unique(config::delete_bitmap_agg_cache_capacity);
+// The default cache is set to 100MB, use memory limit to dynamic
adjustment
+bool is_percent = false;
+int64_t delete_bitmap_agg_cache_cache_limit =
+
ParseUtil::parse_mem_spec(config::delete_bitmap_dynamic_agg_cache_limit,
+ MemInfo::mem_limit(),
MemInfo::physical_mem(), &is_percent);
+_txn_delete_bitmap_cache = std::make_unique(
+delete_bitmap_agg_cache_cache_limit >
config::delete_bitmap_agg_cache_capacity
+? delete_bitmap_agg_cache_cache_limit
+: config::delete_bitmap_agg_cache_capacity);
RETURN_IF_ERROR(_txn_delete_bitmap_cache->init());
_file_cache_block_downloader =
std::make_unique(*this);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](test) disable fault injection to make pipeline task check happy (#38665) [doris]
doris-robot commented on PR #38821: URL: https://github.com/apache/doris/pull/38821#issuecomment-2267253349 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [chore](test) disable fault injection to make pipeline task check happy (#38665) [doris]
sollhui opened a new pull request, #38821: URL: https://github.com/apache/doris/pull/38821 pick (#38665) test_delta_writer_v2_back_pressure_fault_injection would make pipeline task can not finish, disable it temporarily to make pipeline task check happy. ## Proposed changes Issue Number: close #xxx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [log](s3) print detailed error message when failed to access object storage [doris]
github-actions[bot] commented on PR #38585: URL: https://github.com/apache/doris/pull/38585#issuecomment-2267253308 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](test) disable fault injection to make pipeline task check happy (#38665) [doris]
sollhui commented on PR #38821: URL: https://github.com/apache/doris/pull/38821#issuecomment-2267253443 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](file cache) Try to read from remote storage when opening segment with CachedRemoteFileReader [doris]
dataroaring commented on PR #38645: URL: https://github.com/apache/doris/pull/38645#issuecomment-2267254278 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](compaction) reduce the base compaction freeze interval [doris]
github-actions[bot] commented on PR #38780: URL: https://github.com/apache/doris/pull/38780#issuecomment-2267254903 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: [log](s3) print detailed error message when failed to access object storage (#38585)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 3bd9af74e99 [log](s3) print detailed error message when failed to
access object storage (#38585)
3bd9af74e99 is described below
commit 3bd9af74e996fd59b83e6089d8efa9e7f58437e9
Author: Kaijie Chen
AuthorDate: Sun Aug 4 10:21:50 2024 +0800
[log](s3) print detailed error message when failed to access object storage
(#38585)
## Proposed changes
Print detailed error message when failed to access object storage.
---
.../java/org/apache/doris/analysis/CreateStageStmt.java | 12 +++-
.../main/java/org/apache/doris/analysis/LoadStmt.java | 17 +
.../java/org/apache/doris/common/UserException.java | 5 +
.../test_domain_connection_and_ak_sk_correction.groovy | 8
4 files changed, 17 insertions(+), 25 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateStageStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateStageStmt.java
index f777dd995f6..4054584805a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateStageStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateStageStmt.java
@@ -112,16 +112,10 @@ public class CreateStageStmt extends DdlStmt {
remote.headObject("1");
remote.listObjects(null);
} catch (Exception e) {
-LOG.warn("Failed check object storage info={}",
stageProperties.getObjectStoreInfoPB(), e);
-String message = e.getMessage();
-if (message != null) {
-int index = message.indexOf("Error message=");
-if (index != -1) {
-message = message.substring(index);
-}
-}
+LOG.warn("Failed to access object storage, proto={}, err={}",
+stageProperties.getObjectStoreInfoPB(), e.toString());
throw new UserException(InternalErrorCode.GET_REMOTE_DATA_ERROR,
-"Incorrect object storage info, " + message);
+"Failed to access object storage", e);
} finally {
if (remote != null) {
remote.close();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/LoadStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/LoadStmt.java
index a24d1ae72fe..b879bcc7c9a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/LoadStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/LoadStmt.java
@@ -607,14 +607,14 @@ public class LoadStmt extends DdlStmt {
connection.setConnectTimeout(1);
connection.connect();
} catch (Exception e) {
-LOG.warn("Failed to connect endpoint={}", endpoint, e);
-throw new UserException("Incorrect object storage info: " +
e.getMessage());
+LOG.warn("Failed to connect endpoint={}, err={}", endpoint, e);
+throw new UserException("Failed to access object storage", e);
} finally {
if (connection != null) {
try {
connection.disconnect();
} catch (Exception e) {
-LOG.warn("Failed to disconnect connection, endpoint={}",
endpoint, e);
+LOG.warn("Failed to disconnect connection, endpoint={},
err={}", endpoint, e);
}
}
SecurityChecker.getInstance().stopSSRFChecking();
@@ -671,16 +671,9 @@ public class LoadStmt extends DdlStmt {
}
}
} catch (Exception e) {
-LOG.warn("Failed check object info={}", objectInfo, e);
-String message = e.getMessage();
-if (message != null) {
-int index = message.indexOf("Error message=");
-if (index != -1) {
-message = message.substring(index);
-}
-}
+LOG.warn("Failed to access object storage, proto={}, err={}",
objectInfo, e.toString());
throw new UserException(InternalErrorCode.GET_REMOTE_DATA_ERROR,
-"Incorrect object storage info, " + message);
+"Failed to access object storage", e);
} finally {
if (remote != null) {
remote.close();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/UserException.java
b/fe/fe-core/src/main/java/org/apache/doris/common/UserException.java
index 09421e10853..5a2dfb82576 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/UserException.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/UserException.java
@@ -54,7 +54,12 @@ public class UserException extends Exception {
super(Strings.nullToEmpty(msg)
Re: [PR] [log](s3) print detailed error message when failed to access object storage [doris]
dataroaring merged PR #38585: URL: https://github.com/apache/doris/pull/38585 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](compaction) reduce the base compaction freeze interval [doris]
github-actions[bot] commented on PR #38780: URL: https://github.com/apache/doris/pull/38780#issuecomment-2267254926 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (6619b64606f -> e51559516ae)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 6619b64606f [Enhancement](merge-on-write) Support dynamic delete bitmap txn_delete_bitmap_cache (#38761) add e51559516ae [log](s3) print detailed error message when failed to access object storage (#38585) No new revisions were added by this update. Summary of changes: .../java/org/apache/doris/analysis/CreateStageStmt.java | 12 +++- .../main/java/org/apache/doris/analysis/LoadStmt.java | 17 + .../java/org/apache/doris/common/UserException.java | 5 + .../test_domain_connection_and_ak_sk_correction.groovy | 8 4 files changed, 17 insertions(+), 25 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compaction) fix the longest continuous rowsets cannot be selected when missing rowsets [doris]
github-actions[bot] commented on PR #38728: URL: https://github.com/apache/doris/pull/38728#issuecomment-2267256787 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](S3) Refactor S3 file writer's duplicate code into common function [doris]
dataroaring commented on PR #38771: URL: https://github.com/apache/doris/pull/38771#issuecomment-2267257098 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](testcase) add order by to fix unstable output of passwordLeaked [doris]
github-actions[bot] commented on PR #38813: URL: https://github.com/apache/doris/pull/38813#issuecomment-2267258051 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](testcase) add order by to fix unstable output of passwordLeaked [doris]
github-actions[bot] commented on PR #38813: URL: https://github.com/apache/doris/pull/38813#issuecomment-2267258057 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](regression) fix test_primary_key_simple_case [doris]
dataroaring merged PR #38798: URL: https://github.com/apache/doris/pull/38798 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](regression) fix test_primary_key_simple_case (#38798)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8b80fb84151 [fix](regression) fix test_primary_key_simple_case (#38798)
8b80fb84151 is described below
commit 8b80fb841519f2036e0e3fd4a2629aae6c29f476
Author: meiyi
AuthorDate: Sun Aug 4 10:32:02 2024 +0800
[fix](regression) fix test_primary_key_simple_case (#38798)
## Proposed changes
the case is failed:
```
assertTrue(result.size() == 7)
assertTrue(result[6][10] == 25)
^^ERROR LINE^^
Exception:
java.lang.NullPointerException: Cannot invoke method getAt() on null object
```
---
.../suites/unique_with_mow_c_p0/test_primary_key_simple_case.groovy | 2 +-
.../suites/unique_with_mow_p0/test_primary_key_simple_case.groovy | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git
a/regression-test/suites/unique_with_mow_c_p0/test_primary_key_simple_case.groovy
b/regression-test/suites/unique_with_mow_c_p0/test_primary_key_simple_case.groovy
index 49d4cd53ddc..94b3051cba3 100644
---
a/regression-test/suites/unique_with_mow_c_p0/test_primary_key_simple_case.groovy
+++
b/regression-test/suites/unique_with_mow_c_p0/test_primary_key_simple_case.groovy
@@ -66,7 +66,7 @@ suite("test_primary_key_simple_case") {
(5, '2017-10-01', 'Beijing', 10, 1, NULL, NULL, '2020-01-05', 1,
34, 20)
"""
-result = sql """ SELECT * FROM ${tableName} t ORDER BY user_id; """
+def result = sql """ SELECT * FROM ${tableName} t ORDER BY user_id; """
assertTrue(result.size() == 5)
assertTrue(result[0].size() == 11)
diff --git
a/regression-test/suites/unique_with_mow_p0/test_primary_key_simple_case.groovy
b/regression-test/suites/unique_with_mow_p0/test_primary_key_simple_case.groovy
index a2e351372e3..91d1da9162c 100644
---
a/regression-test/suites/unique_with_mow_p0/test_primary_key_simple_case.groovy
+++
b/regression-test/suites/unique_with_mow_p0/test_primary_key_simple_case.groovy
@@ -62,7 +62,7 @@ suite("test_primary_key_simple_case") {
(5, '2017-10-01', 'Beijing', 10, 1, NULL, NULL, '2020-01-05', 1,
34, 20)
"""
-result = sql """ SELECT * FROM ${tableName} t ORDER BY user_id; """
+def result = sql """ SELECT * FROM ${tableName} t ORDER BY user_id; """
assertTrue(result.size() == 5)
assertTrue(result[0].size() == 11)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement]use awaitility.await() replace thread.sleep(), neredis part [doris]
github-actions[bot] commented on PR #38816: URL: https://github.com/apache/doris/pull/38816#issuecomment-2267258734 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement]use awaitility.await() replace thread.sleep(), neredis part [doris]
github-actions[bot] commented on PR #38816: URL: https://github.com/apache/doris/pull/38816#issuecomment-2267258743 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](txn) fix abortTxn by label does not acquire table write lock [doris]
dataroaring merged PR #38777: URL: https://github.com/apache/doris/pull/38777 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](txn) fix abortTxn by label does not acquire table write lock (#38777)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new b7ff97e6f51 [fix](txn) fix abortTxn by label does not acquire table
write lock (#38777)
b7ff97e6f51 is described below
commit b7ff97e6f51de74ed651e253472a75080d954e7b
Author: meiyi
AuthorDate: Sun Aug 4 10:32:50 2024 +0800
[fix](txn) fix abortTxn by label does not acquire table write lock (#38777)
---
.../java/org/apache/doris/transaction/GlobalTransactionMgr.java| 7 +--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java
index 57ee12fe86c..f97fbf5f191 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java
@@ -383,8 +383,11 @@ public class GlobalTransactionMgr implements
GlobalTransactionMgrIface {
// for http cancel stream load api
@Override
public void abortTransaction(Long dbId, String label, String reason)
throws UserException {
-DatabaseTransactionMgr dbTransactionMgr =
getDatabaseTransactionMgr(dbId);
-dbTransactionMgr.abortTransaction(label, reason);
+Long txnId = getTransactionId(dbId, label);
+if (txnId == null) {
+throw new AnalysisException("txn with label " + label + " does not
exist");
+}
+abortTransaction(dbId, txnId, reason);
}
@Override
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] add information_schema.table_properties #38745 [doris]
dataroaring commented on PR #38746: URL: https://github.com/apache/doris/pull/38746#issuecomment-2267259872 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] add information_schema.table_properties #38745 [doris]
github-actions[bot] commented on PR #38746: URL: https://github.com/apache/doris/pull/38746#issuecomment-2267260137 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement] add information_schema.table_properties #38745 [doris]
github-actions[bot] commented on PR #38746: URL: https://github.com/apache/doris/pull/38746#issuecomment-2267260141 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](gson) Fix Expr deserialize compatibility [doris]
dataroaring merged PR #38799: URL: https://github.com/apache/doris/pull/38799 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](gson) Fix Expr deserialize compatibility (#38799)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 0b61263e2f7 [fix](gson) Fix Expr deserialize compatibility (#38799)
0b61263e2f7 is described below
commit 0b61263e2f7c44c75377a1120c20a024990074c6
Author: walter
AuthorDate: Sun Aug 4 10:36:39 2024 +0800
[fix](gson) Fix Expr deserialize compatibility (#38799)
Since GsonBuilder.create() adds all registered factories to GSON in
reverse order, and ExprAdapterFactory is registered before the
RuntimeTypeAdapterFactory for Expr, ExprAdapterFactory will not be
executed.
This PR adjusts their registration order. Now, it will first check-in
ExprAdapterFactory whether to use the pre-134 deserialize method and
then attempt to use the RuntimeTypeAdapterFactory for Expr.class.
---
.../org/apache/doris/persist/gson/GsonUtils.java | 12 -
.../apache/doris/persist/gson/GsonUtils134.java| 12 -
.../java/org/apache/doris/persist/ExprTest.java| 52 ++
3 files changed, 74 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
index 9f365808a6f..0a6043ac685 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
@@ -579,6 +579,11 @@ public class GsonUtils {
// the builder of GSON instance.
// Add any other adapters if necessary.
+//
+// ATTN:
+// Since GsonBuilder.create() adds all registered factories to GSON in
reverse order, if you
+// need to ensure the search order of two RuntimeTypeAdapterFactory
instances, be sure to
+// register them in reverse priority order.
private static final GsonBuilder GSON_BUILDER = new GsonBuilder()
.setObjectToNumberStrategy(ToNumberPolicy.LONG_OR_DOUBLE)
.addSerializationExclusionStrategy(
@@ -589,8 +594,8 @@ public class GsonUtils {
.registerTypeHierarchyAdapter(Multimap.class, new
GuavaMultimapAdapter())
.registerTypeAdapterFactory(new PostProcessTypeAdapterFactory())
.registerTypeAdapterFactory(new PreProcessTypeAdapterFactory())
-.registerTypeAdapterFactory(new ExprAdapterFactory())
.registerTypeAdapterFactory(exprAdapterFactory)
+.registerTypeAdapterFactory(new ExprAdapterFactory())
.registerTypeAdapterFactory(columnTypeAdapterFactory)
.registerTypeAdapterFactory(distributionInfoTypeAdapterFactory)
.registerTypeAdapterFactory(resourceTypeAdapterFactory)
@@ -776,6 +781,11 @@ public class GsonUtils {
final Class rawType = (Class) type.getRawType();
final TypeAdapter delegate = gson.getDelegateAdapter(this,
type);
+if (!Expr.class.isAssignableFrom(rawType)) {
+// reduce the stack depth.
+return null;
+}
+
return new TypeAdapter() {
public void write(JsonWriter out, T value) throws IOException {
delegate.write(out, value);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils134.java
b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils134.java
index 1359bb170df..147c403869e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils134.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils134.java
@@ -473,6 +473,11 @@ public class GsonUtils134 {
// the builder of GSON instance.
// Add any other adapters if necessary.
+//
+// ATTN:
+// Since GsonBuilder.create() adds all registered factories to GSON in
reverse order, if you
+// need to ensure the search order of two RuntimeTypeAdapterFactory
instances, be sure to
+// register them in reverse priority order.
private static final GsonBuilder GSON_BUILDER = new
GsonBuilder().addSerializationExclusionStrategy(
new
HiddenAnnotationExclusionStrategy()).enableComplexMapKeySerialization()
.addReflectionAccessFilter(ReflectionAccessFilter.BLOCK_INACCESSIBLE_JAVA)
@@ -480,8 +485,8 @@ public class GsonUtils134 {
// .registerTypeHierarchyAdapter(Expr.class, new ExprAdapter())
.registerTypeHierarchyAdapter(Multimap.class, new
GuavaMultimapAdapter())
.registerTypeAdapterFactory(new PostProcessTypeAdapterFactory())
-.registerTypeAdapterFactory(new ExprAdapterFactory())
.registerTypeAdapterFactory(exprAdapterFactory)
+.registerTypeAdapterFactory(new ExprAdapterFactory())
.registerTypeAdapterFactory(columnTypeAdapterFactory)
.regi
(doris) branch branch-3.0 updated (3bd9af74e99 -> 5f67a85f817)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git from 3bd9af74e99 [log](s3) print detailed error message when failed to access object storage (#38585) new 699e6c3b4f8 [fix](regression) fix test_primary_key_simple_case (#38798) new 05e0274030b [fix](txn) fix abortTxn by label does not acquire table write lock (#38777) new 5f67a85f817 [fix](gson) Fix Expr deserialize compatibility (#38799) The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/doris/persist/gson/GsonUtils.java | 12 - .../apache/doris/persist/gson/GsonUtils134.java| 12 - .../doris/transaction/GlobalTransactionMgr.java| 7 ++- .../java/org/apache/doris/persist/ExprTest.java| 52 ++ .../test_primary_key_simple_case.groovy| 2 +- .../test_primary_key_simple_case.groovy| 2 +- 6 files changed, 81 insertions(+), 6 deletions(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/persist/ExprTest.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/03: [fix](regression) fix test_primary_key_simple_case (#38798)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 699e6c3b4f80bebafc57dfc261957c10411f0eae
Author: meiyi
AuthorDate: Sun Aug 4 10:32:02 2024 +0800
[fix](regression) fix test_primary_key_simple_case (#38798)
## Proposed changes
the case is failed:
```
assertTrue(result.size() == 7)
assertTrue(result[6][10] == 25)
^^ERROR LINE^^
Exception:
java.lang.NullPointerException: Cannot invoke method getAt() on null object
```
---
.../unique_with_mow_p0/cluster_key/test_primary_key_simple_case.groovy | 2 +-
.../suites/unique_with_mow_p0/test_primary_key_simple_case.groovy | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git
a/regression-test/suites/unique_with_mow_p0/cluster_key/test_primary_key_simple_case.groovy
b/regression-test/suites/unique_with_mow_p0/cluster_key/test_primary_key_simple_case.groovy
index 788f934fdf0..39efc150e7f 100644
---
a/regression-test/suites/unique_with_mow_p0/cluster_key/test_primary_key_simple_case.groovy
+++
b/regression-test/suites/unique_with_mow_p0/cluster_key/test_primary_key_simple_case.groovy
@@ -66,7 +66,7 @@ suite("test_primary_key_simple_case") {
(5, '2017-10-01', 'Beijing', 10, 1, NULL, NULL, '2020-01-05', 1,
34, 20)
"""
-result = sql """ SELECT * FROM ${tableName} t ORDER BY user_id; """
+def result = sql """ SELECT * FROM ${tableName} t ORDER BY user_id; """
assertTrue(result.size() == 5)
assertTrue(result[0].size() == 11)
diff --git
a/regression-test/suites/unique_with_mow_p0/test_primary_key_simple_case.groovy
b/regression-test/suites/unique_with_mow_p0/test_primary_key_simple_case.groovy
index a2e351372e3..91d1da9162c 100644
---
a/regression-test/suites/unique_with_mow_p0/test_primary_key_simple_case.groovy
+++
b/regression-test/suites/unique_with_mow_p0/test_primary_key_simple_case.groovy
@@ -62,7 +62,7 @@ suite("test_primary_key_simple_case") {
(5, '2017-10-01', 'Beijing', 10, 1, NULL, NULL, '2020-01-05', 1,
34, 20)
"""
-result = sql """ SELECT * FROM ${tableName} t ORDER BY user_id; """
+def result = sql """ SELECT * FROM ${tableName} t ORDER BY user_id; """
assertTrue(result.size() == 5)
assertTrue(result[0].size() == 11)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 02/03: [fix](txn) fix abortTxn by label does not acquire table write lock (#38777)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 05e0274030b03e3dd75c80eb053f52613ba6d462
Author: meiyi
AuthorDate: Sun Aug 4 10:32:50 2024 +0800
[fix](txn) fix abortTxn by label does not acquire table write lock (#38777)
---
.../java/org/apache/doris/transaction/GlobalTransactionMgr.java| 7 +--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java
b/fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java
index 71b8094..b4373a26ea6 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java
@@ -381,8 +381,11 @@ public class GlobalTransactionMgr implements
GlobalTransactionMgrIface {
// for http cancel stream load api
@Override
public void abortTransaction(Long dbId, String label, String reason)
throws UserException {
-DatabaseTransactionMgr dbTransactionMgr =
getDatabaseTransactionMgr(dbId);
-dbTransactionMgr.abortTransaction(label, reason);
+Long txnId = getTransactionId(dbId, label);
+if (txnId == null) {
+throw new AnalysisException("txn with label " + label + " does not
exist");
+}
+abortTransaction(dbId, txnId, reason);
}
@Override
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 03/03: [fix](gson) Fix Expr deserialize compatibility (#38799)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5f67a85f817f176a96023515a1e834e5a7630745
Author: walter
AuthorDate: Sun Aug 4 10:36:39 2024 +0800
[fix](gson) Fix Expr deserialize compatibility (#38799)
Since GsonBuilder.create() adds all registered factories to GSON in
reverse order, and ExprAdapterFactory is registered before the
RuntimeTypeAdapterFactory for Expr, ExprAdapterFactory will not be
executed.
This PR adjusts their registration order. Now, it will first check-in
ExprAdapterFactory whether to use the pre-134 deserialize method and
then attempt to use the RuntimeTypeAdapterFactory for Expr.class.
---
.../org/apache/doris/persist/gson/GsonUtils.java | 12 -
.../apache/doris/persist/gson/GsonUtils134.java| 12 -
.../java/org/apache/doris/persist/ExprTest.java| 52 ++
3 files changed, 74 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
index 9f365808a6f..0a6043ac685 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java
@@ -579,6 +579,11 @@ public class GsonUtils {
// the builder of GSON instance.
// Add any other adapters if necessary.
+//
+// ATTN:
+// Since GsonBuilder.create() adds all registered factories to GSON in
reverse order, if you
+// need to ensure the search order of two RuntimeTypeAdapterFactory
instances, be sure to
+// register them in reverse priority order.
private static final GsonBuilder GSON_BUILDER = new GsonBuilder()
.setObjectToNumberStrategy(ToNumberPolicy.LONG_OR_DOUBLE)
.addSerializationExclusionStrategy(
@@ -589,8 +594,8 @@ public class GsonUtils {
.registerTypeHierarchyAdapter(Multimap.class, new
GuavaMultimapAdapter())
.registerTypeAdapterFactory(new PostProcessTypeAdapterFactory())
.registerTypeAdapterFactory(new PreProcessTypeAdapterFactory())
-.registerTypeAdapterFactory(new ExprAdapterFactory())
.registerTypeAdapterFactory(exprAdapterFactory)
+.registerTypeAdapterFactory(new ExprAdapterFactory())
.registerTypeAdapterFactory(columnTypeAdapterFactory)
.registerTypeAdapterFactory(distributionInfoTypeAdapterFactory)
.registerTypeAdapterFactory(resourceTypeAdapterFactory)
@@ -776,6 +781,11 @@ public class GsonUtils {
final Class rawType = (Class) type.getRawType();
final TypeAdapter delegate = gson.getDelegateAdapter(this,
type);
+if (!Expr.class.isAssignableFrom(rawType)) {
+// reduce the stack depth.
+return null;
+}
+
return new TypeAdapter() {
public void write(JsonWriter out, T value) throws IOException {
delegate.write(out, value);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils134.java
b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils134.java
index 1359bb170df..147c403869e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils134.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils134.java
@@ -473,6 +473,11 @@ public class GsonUtils134 {
// the builder of GSON instance.
// Add any other adapters if necessary.
+//
+// ATTN:
+// Since GsonBuilder.create() adds all registered factories to GSON in
reverse order, if you
+// need to ensure the search order of two RuntimeTypeAdapterFactory
instances, be sure to
+// register them in reverse priority order.
private static final GsonBuilder GSON_BUILDER = new
GsonBuilder().addSerializationExclusionStrategy(
new
HiddenAnnotationExclusionStrategy()).enableComplexMapKeySerialization()
.addReflectionAccessFilter(ReflectionAccessFilter.BLOCK_INACCESSIBLE_JAVA)
@@ -480,8 +485,8 @@ public class GsonUtils134 {
// .registerTypeHierarchyAdapter(Expr.class, new ExprAdapter())
.registerTypeHierarchyAdapter(Multimap.class, new
GuavaMultimapAdapter())
.registerTypeAdapterFactory(new PostProcessTypeAdapterFactory())
-.registerTypeAdapterFactory(new ExprAdapterFactory())
.registerTypeAdapterFactory(exprAdapterFactory)
+.registerTypeAdapterFactory(new ExprAdapterFactory())
.registerTypeAdapterFactory(columnTypeAdapterFactory)
.registerTypeAdapterFactory(distributionInfoTypeAdapterFactory)
.registerTypeAdapterFactory(resourceTypeAdapterFactory)
@@ -660,6 +665,11 @@ public class GsonUtils134 {
Re: [PR] [fix](fe) Fix the default value of ReplacePartitionClause.isStrictRange [doris]
dataroaring merged PR #38688: URL: https://github.com/apache/doris/pull/38688 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (0b61263e2f7 -> 312c7e1e623)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 0b61263e2f7 [fix](gson) Fix Expr deserialize compatibility (#38799) add 312c7e1e623 [fix](fe) Fix the default value of ReplacePartitionClause.isStrictRange (#38688) No new revisions were added by this update. Summary of changes: .../apache/doris/analysis/ReplacePartitionClause.java | 17 + 1 file changed, 17 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](compaction) reduce the base compaction freeze interval [doris]
dataroaring merged PR #38780: URL: https://github.com/apache/doris/pull/38780 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (312c7e1e623 -> 38dbfa821c1)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 312c7e1e623 [fix](fe) Fix the default value of ReplacePartitionClause.isStrictRange (#38688) add 38dbfa821c1 [enhancement](compaction) reduce the base compaction freeze interval (#38780) No new revisions were added by this update. Summary of changes: be/src/cloud/config.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: [enhancement](compaction) reduce the base compaction freeze interval (#38780)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 8e55276773a [enhancement](compaction) reduce the base compaction freeze interval (#38780) 8e55276773a is described below commit 8e55276773ac7790d7b4418f4aa100ee841ec500 Author: Luwei <814383...@qq.com> AuthorDate: Sun Aug 4 10:39:48 2024 +0800 [enhancement](compaction) reduce the base compaction freeze interval (#38780) --- be/src/cloud/config.cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/be/src/cloud/config.cpp b/be/src/cloud/config.cpp index 902502e0aac..82c466120e9 100644 --- a/be/src/cloud/config.cpp +++ b/be/src/cloud/config.cpp @@ -35,7 +35,7 @@ DEFINE_Int64(tablet_cache_shards, "16"); DEFINE_mInt32(tablet_sync_interval_s, "1800"); DEFINE_mInt64(min_compaction_failure_interval_ms, "5000"); -DEFINE_mInt64(base_compaction_freeze_interval_s, "86400"); +DEFINE_mInt64(base_compaction_freeze_interval_s, "7200"); DEFINE_mInt64(cu_compaction_freeze_interval_s, "1200"); DEFINE_mInt64(cumu_compaction_interval_s, "1800"); - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compaction) disable cumu delete rowset from performing ordered data compaction [doris]
dataroaring merged PR #38742: URL: https://github.com/apache/doris/pull/38742 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](compaction) disable cumu delete rowset from performing ordered data compaction (#38742)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new dfdec4d161b [fix](compaction) disable cumu delete rowset from
performing ordered data compaction (#38742)
dfdec4d161b is described below
commit dfdec4d161b20f21e6beb4239a23816d387b382c
Author: Luwei <814383...@qq.com>
AuthorDate: Sun Aug 4 10:40:32 2024 +0800
[fix](compaction) disable cumu delete rowset from performing ordered data
compaction (#38742)
## problem
When enable_delete_when_cumu_compaction = true, multiple delete rowsets
may be merged into an empty data rowset through ordered data compaction,
causing the delete predicate to be lost.
---
be/src/olap/compaction.cpp | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/be/src/olap/compaction.cpp b/be/src/olap/compaction.cpp
index 2190521589c..935c9991e97 100644
--- a/be/src/olap/compaction.cpp
+++ b/be/src/olap/compaction.cpp
@@ -352,7 +352,9 @@ bool CompactionMixin::handle_ordered_data_compaction() {
// check delete version: if compaction type is base compaction and
// has a delete version, use original compaction
-if (compaction_type() == ReaderType::READER_BASE_COMPACTION) {
+if (compaction_type() == ReaderType::READER_BASE_COMPACTION ||
+(_allow_delete_in_cumu_compaction &&
+ compaction_type() == ReaderType::READER_CUMULATIVE_COMPACTION)) {
for (auto& rowset : _input_rowsets) {
if (rowset->rowset_meta()->has_delete_predicate()) {
return false;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-3.0 updated: [fix](compaction) disable cumu delete rowset from performing ordered data compaction (#38742)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 51f2e352eb3 [fix](compaction) disable cumu delete rowset from
performing ordered data compaction (#38742)
51f2e352eb3 is described below
commit 51f2e352eb34114395254850037298b879f2de4a
Author: Luwei <814383...@qq.com>
AuthorDate: Sun Aug 4 10:40:32 2024 +0800
[fix](compaction) disable cumu delete rowset from performing ordered data
compaction (#38742)
## problem
When enable_delete_when_cumu_compaction = true, multiple delete rowsets
may be merged into an empty data rowset through ordered data compaction,
causing the delete predicate to be lost.
---
be/src/olap/compaction.cpp | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/be/src/olap/compaction.cpp b/be/src/olap/compaction.cpp
index dc4dd419a17..1d0a592e535 100644
--- a/be/src/olap/compaction.cpp
+++ b/be/src/olap/compaction.cpp
@@ -349,7 +349,9 @@ bool CompactionMixin::handle_ordered_data_compaction() {
// check delete version: if compaction type is base compaction and
// has a delete version, use original compaction
-if (compaction_type() == ReaderType::READER_BASE_COMPACTION) {
+if (compaction_type() == ReaderType::READER_BASE_COMPACTION ||
+(_allow_delete_in_cumu_compaction &&
+ compaction_type() == ReaderType::READER_CUMULATIVE_COMPACTION)) {
for (auto& rowset : _input_rowsets) {
if (rowset->rowset_meta()->has_delete_predicate()) {
return false;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compaction) fix the longest continuous rowsets cannot be selected when missing rowsets [doris]
dataroaring merged PR #38728: URL: https://github.com/apache/doris/pull/38728 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](compaction) fix the longest continuous rowsets cannot be selected when missing rowsets (#38728)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 4c338398f25 [fix](compaction) fix the longest continuous rowsets
cannot be selected when missing rowsets (#38728)
4c338398f25 is described below
commit 4c338398f2557fe5aadde76539b6c577d80c811d
Author: Luwei <814383...@qq.com>
AuthorDate: Sun Aug 4 10:41:50 2024 +0800
[fix](compaction) fix the longest continuous rowsets cannot be selected
when missing rowsets (#38728)
### problem
When version is missing, the code for selecting the longest continuous
version has a bug. Only the version before the missing version will be
selected.
For example: the current version is version [1-1], version [2-2],
version [4-4], version [5-5], version [6-6], version [7-7], and version
[3-3] is missing.
The current result is to return version [1-1], version [2-2] instead of
version [4-4], version [5-5], version [6-6], version [7-7]
---
be/src/olap/cumulative_compaction.cpp | 34 ++--
be/src/olap/cumulative_compaction.h | 3 +
be/test/olap/cumulative_compaction_test.cpp | 266
3 files changed, 291 insertions(+), 12 deletions(-)
diff --git a/be/src/olap/cumulative_compaction.cpp
b/be/src/olap/cumulative_compaction.cpp
index 2c7e654787a..5b72ad0ce70 100644
--- a/be/src/olap/cumulative_compaction.cpp
+++ b/be/src/olap/cumulative_compaction.cpp
@@ -35,16 +35,19 @@
namespace doris {
using namespace ErrorCode;
-namespace {
-
-void find_longest_consecutive_version(std::vector* rowsets,
- std::vector* missing_version) {
+void
CumulativeCompaction::find_longest_consecutive_version(std::vector*
rowsets,
+
std::vector* missing_version) {
if (rowsets->empty()) {
return;
}
RowsetSharedPtr prev_rowset = rowsets->front();
size_t i = 1;
+int max_start = 0;
+int max_length = 1;
+
+int start = 0;
+int length = 1;
for (; i < rowsets->size(); ++i) {
RowsetSharedPtr rowset = (*rowsets)[i];
if (rowset->start_version() != prev_rowset->end_version() + 1) {
@@ -52,16 +55,22 @@ void
find_longest_consecutive_version(std::vector* rowsets,
missing_version->push_back(prev_rowset->version());
missing_version->push_back(rowset->version());
}
-break;
+start = i;
+length = 1;
+} else {
+length++;
+}
+
+if (length > max_length) {
+max_start = start;
+max_length = length;
}
+
prev_rowset = rowset;
}
-
-rowsets->resize(i);
+*rowsets = {rowsets->begin() + max_start, rowsets->begin() + max_start +
max_length};
}
-} // namespace
-
CumulativeCompaction::CumulativeCompaction(StorageEngine& engine, const
TabletSharedPtr& tablet)
: CompactionMixin(engine, tablet,
"CumulativeCompaction:" +
std::to_string(tablet->tablet_id())) {}
@@ -127,10 +136,11 @@ Status CumulativeCompaction::pick_rowsets_to_compact() {
std::vector missing_versions;
find_longest_consecutive_version(&candidate_rowsets, &missing_versions);
if (!missing_versions.empty()) {
-DCHECK(missing_versions.size() == 2);
+DCHECK(missing_versions.size() % 2 == 0);
LOG(WARNING) << "There are missed versions among rowsets. "
- << "prev rowset verison=" << missing_versions[0]
- << ", next rowset version=" << missing_versions[1]
+ << "total missed version size: " <<
missing_versions.size() / 2
+ << " first missed version prev rowset verison=" <<
missing_versions[0]
+ << ", first missed version next rowset version=" <<
missing_versions[1]
<< ", tablet=" << _tablet->tablet_id();
}
diff --git a/be/src/olap/cumulative_compaction.h
b/be/src/olap/cumulative_compaction.h
index 14527bf2fab..276e3b34903 100644
--- a/be/src/olap/cumulative_compaction.h
+++ b/be/src/olap/cumulative_compaction.h
@@ -44,6 +44,9 @@ private:
Status pick_rowsets_to_compact();
+void find_longest_consecutive_version(std::vector*
rowsets,
+ std::vector*
missing_version);
+
Version _last_delete_version {-1, -1};
};
diff --git a/be/test/olap/cumulative_compaction_test.cpp
b/be/test/olap/cumulative_compaction_test.cpp
new file mode 100644
index 000..11258d8567d
--- /dev/null
+++ b/be/test/olap/cumulative_compaction_test.cpp
@@ -0,0 +1,266 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with t
(doris) branch master updated (4c338398f25 -> 3d233891f2f)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 4c338398f25 [fix](compaction) fix the longest continuous rowsets cannot be selected when missing rowsets (#38728) add 3d233891f2f [Fix](load) The value of the index id printed in the log is incorrect (#38790) No new revisions were added by this update. Summary of changes: be/src/runtime/tablets_channel.cpp | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](load) The value of the index id printed in the log is incorrect [doris]
dataroaring merged PR #38790: URL: https://github.com/apache/doris/pull/38790 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
