Re: [PR] [Refactor](dialect) Add sql dialect converter plugins [doris]
dutyu commented on PR #28890: URL: https://github.com/apache/doris/pull/28890#issuecomment-1868235259 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [opt](information_schema) support information_schema in external catalog [doris]
morningman opened a new pull request, #28919: URL: https://github.com/apache/doris/pull/28919 ## Proposed changes Draft ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](create tablet) default create tablet round robin [doris]
yujun777 commented on PR #28911: URL: https://github.com/apache/doris/pull/28911#issuecomment-1868238982 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Refactor](dialect) Add sql dialect converter plugins [doris]
doris-robot commented on PR #28890: URL: https://github.com/apache/doris/pull/28890#issuecomment-1868241922 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit 2086add8ec732822b5c32ac2344748500a36f784, data reload: false run tpch-sf100 query with default conf and session variables q1 4685442844194419 q2 373 146 158 146 q3 1460126612371237 q4 886 903 886 q5 3168313331773133 q6 247 129 129 129 q7 986 486 492 486 q8 2171222521852185 q9 6677666766676667 q10 3226325432663254 q11 308 186 192 186 q12 348 212 206 206 q13 4535380937963796 q14 244 210 211 210 q15 561 521 521 521 q16 436 384 381 381 q17 1002620 526 526 q18 7035675868506758 q19 1532143914071407 q20 524 319 314 314 q21 3075264226312631 q22 348 280 283 280 Total cold run time: 44052 ms Total hot run time: 39758 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4332439243284328 q2 269 165 170 165 q3 3502349934963496 q4 2393235723682357 q5 5694573757115711 q6 239 122 123 122 q7 2356187218771872 q8 3518351535073507 q9 9034899089778977 q10 3915400339973997 q11 488 373 368 368 q12 761 579 599 579 q13 4266358135763576 q14 288 250 255 250 q15 581 520 519 519 q16 503 466 447 447 q17 1888186818491849 q18 8499813581508135 q19 1738175917561756 q20 2269193719421937 q21 6490614461316131 q22 499 414 407 407 Total cold run time: 63522 ms Total hot run time: 60486 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](create tablet) default create tablet round robin [doris]
doris-robot commented on PR #28911: URL: https://github.com/apache/doris/pull/28911#issuecomment-1868243010 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit 8719a14529dff4ce5d3e27e197da3634e1917613, data reload: false run tpch-sf100 query with default conf and session variables q1 4754441844294418 q2 368 147 158 147 q3 1469133512301230 q4 1118892 879 879 q5 3182315231823152 q6 250 131 133 131 q7 990 489 488 488 q8 2158222022012201 q9 6657666166696661 q10 3232325132833251 q11 310 186 182 182 q12 349 211 202 202 q13 4570382237813781 q14 244 212 217 212 q15 577 528 528 528 q16 451 382 385 382 q17 999 590 524 524 q18 7000682967416741 q19 1516143514101410 q20 504 295 313 295 q21 3132264426672644 q22 350 279 282 279 Total cold run time: 44180 ms Total hot run time: 39738 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4339432843304328 q2 269 168 168 168 q3 3480349534743474 q4 2401236423552355 q5 5705570057135700 q6 242 126 125 125 q7 2377187218401840 q8 3516352935233523 q9 8977899089688968 q10 3937400039933993 q11 488 377 360 360 q12 759 587 594 587 q13 4318356935633563 q14 285 255 250 250 q15 575 520 515 515 q16 502 442 477 442 q17 1880186618421842 q18 8509801482158014 q19 1758173717491737 q20 2251195419401940 q21 6517614661416141 q22 507 427 427 427 Total cold run time: 63592 ms Total hot run time: 60292 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](execute) optimize for ColumnNullable's serialize_vec/deserialize_vec [doris]
BiteThet commented on PR #28788: URL: https://github.com/apache/doris/pull/28788#issuecomment-1868243699 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Refactor](dialect) Add sql dialect converter plugins [doris]
doris-robot commented on PR #28890: URL: https://github.com/apache/doris/pull/28890#issuecomment-1868244056 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.45 seconds stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.0 seconds inserted 1000 Rows, about 344K ops/s storage size: 17183825534 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](create tablet) default create tablet round robin [doris]
doris-robot commented on PR #28911: URL: https://github.com/apache/doris/pull/28911#issuecomment-1868244574 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.62 seconds stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.6 seconds inserted 1000 Rows, about 349K ops/s storage size: 17183790601 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](execute) optimize for ColumnNullable's serialize_vec/deserialize_vec [doris]
doris-robot commented on PR #28788: URL: https://github.com/apache/doris/pull/28788#issuecomment-1868247543 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit 0487a9b89fbd3ac311fa22b84111c3b435375abc, data reload: false run tpch-sf100 query with default conf and session variables q1 4703441244834412 q2 371 144 159 144 q3 1448126712171217 q4 1105900 844 844 q5 3157315631473147 q6 249 132 127 127 q7 979 491 475 475 q8 2169221521692169 q9 6691670466476647 q10 3227327432723272 q11 304 183 186 183 q12 346 212 203 203 q13 4571379037623762 q14 252 212 209 209 q15 565 529 523 523 q16 440 390 386 386 q17 1030633 548 548 q18 7089689268136813 q19 1517137214811372 q20 550 301 288 288 q21 3085264526352635 q22 348 282 282 282 Total cold run time: 44196 ms Total hot run time: 39658 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4353437043684368 q2 267 163 176 163 q3 3502349434863486 q4 2387236723652365 q5 5719571557135713 q6 238 124 123 123 q7 2357184718871847 q8 3510352335233523 q9 9032901789518951 q10 3905400340134003 q11 481 360 370 360 q12 763 591 608 591 q13 4310358635763576 q14 287 264 248 248 q15 575 519 522 519 q16 500 451 477 451 q17 1895187118711871 q18 8441813881128112 q19 1725178217121712 q20 2252192819341928 q21 6473612961576129 q22 497 448 411 411 Total cold run time: 63469 ms Total hot run time: 60450 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](create tablet) default create tablet round robin [doris]
yujun777 commented on PR #28911: URL: https://github.com/apache/doris/pull/28911#issuecomment-1868247799 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](bulk-load) cancel loading tasks directly without retrying when timeout exceeded [doris]
github-actions[bot] commented on PR #28666: URL: https://github.com/apache/doris/pull/28666#issuecomment-1868249411 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet) the end offset of column chunk may be wrong in parquet metadata [doris]
AshinGau commented on PR #28891: URL: https://github.com/apache/doris/pull/28891#issuecomment-1868249505 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet) the end offset of column chunk may be wrong in parquet metadata [doris]
AshinGau commented on PR #28893: URL: https://github.com/apache/doris/pull/28893#issuecomment-1868249791 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](execute) optimize for ColumnNullable's serialize_vec/deserialize_vec [doris]
doris-robot commented on PR #28788: URL: https://github.com/apache/doris/pull/28788#issuecomment-1868250024 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.41 seconds stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.6 seconds inserted 1000 Rows, about 349K ops/s storage size: 17183821545 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet) the end offset of column chunk may be wrong in parquet metadata [doris]
github-actions[bot] commented on PR #28891: URL: https://github.com/apache/doris/pull/28891#issuecomment-1868250588 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet) the end offset of column chunk may be wrong in parquet metadata [doris]
github-actions[bot] commented on PR #28893: URL: https://github.com/apache/doris/pull/28893#issuecomment-1868250746 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.0-var](disk balance) Impr disk rebalancer sched #26412 [doris]
yujun777 commented on PR #28920: URL: https://github.com/apache/doris/pull/28920#issuecomment-1868250789 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [branch-2.0-var](disk balance) Impr disk rebalancer sched #26412 [doris]
yujun777 opened a new pull request, #28920: URL: https://github.com/apache/doris/pull/28920 pick: #26412 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](chore) update DCHECK to avoid core during stress test [doris]
github-actions[bot] commented on PR #28895: URL: https://github.com/apache/doris/pull/28895#issuecomment-1868251025 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](page cache) insert into setting to disable page cache [doris]
yiguolei commented on code in PR #28913: URL: https://github.com/apache/doris/pull/28913#discussion_r1435546767 ## fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/InsertIntoTableCommand.java: ## @@ -102,6 +102,8 @@ public void setJobId(long jobId) { @Override public void run(ConnectContext ctx, StmtExecutor executor) throws Exception { +// insert into setting to disable page cache +ctx.getSessionVariable().enablePageCache = false; Review Comment: If you modify the code like this. when the insert into select command finished. and user want to execute select command, the page cache value is false too. The performance maybe bad. I think you could refer the code insert into xxx select /* SET_VAR=xxx*/ to know how to set the session variable at sql level not at session level. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.0-var](disk balance) Impr disk rebalancer sched #26412 [doris]
yujun777 commented on PR #28920: URL: https://github.com/apache/doris/pull/28920#issuecomment-1868253039 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet) the end offset of column chunk may be wrong in parquet metadata [doris]
doris-robot commented on PR #28891: URL: https://github.com/apache/doris/pull/28891#issuecomment-1868253030 TeamCity be ut coverage result: Function Coverage: 36.59% (8550/23364) Line Coverage: 28.66% (69505/242492) Region Coverage: 27.67% (35958/129943) Branch Coverage: 24.40% (18380/75324) Coverage Report: http://coverage.selectdb-in.cc/coverage/7037d3db0a423f5f471d2503c83b5e6343b03183_7037d3db0a423f5f471d2503c83b5e6343b03183/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet) the end offset of column chunk may be wrong in parquet metadata [doris]
doris-robot commented on PR #28891: URL: https://github.com/apache/doris/pull/28891#issuecomment-1868253642 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit 7037d3db0a423f5f471d2503c83b5e6343b03183, data reload: false run tpch-sf100 query with default conf and session variables q1 4703437244434372 q2 369 147 159 147 q3 1460125312491249 q4 1107889 908 889 q5 3173315631783156 q6 247 136 135 135 q7 981 487 487 487 q8 2176224621912191 q9 6679665066776650 q10 3196327132833271 q11 305 189 178 178 q12 351 211 209 209 q13 4572379438133794 q14 245 211 221 211 q15 575 524 524 524 q16 441 379 392 379 q17 1014674 602 602 q18 7144676967796769 q19 1540143814221422 q20 543 311 289 289 q21 3084265726892657 q22 354 285 289 285 Total cold run time: 44259 ms Total hot run time: 39866 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4338432043444320 q2 274 167 169 167 q3 3514350934893489 q4 2397237823722372 q5 5707570657085706 q6 250 121 125 121 q7 2369187118611861 q8 3523352235403522 q9 9011902089888988 q10 3927401540214015 q11 487 363 364 363 q12 760 616 601 601 q13 4303359835313531 q14 293 261 253 253 q15 570 535 521 521 q16 514 462 507 462 q17 1898185018731850 q18 8581800179797979 q19 1780177017791770 q20 2271195019341934 q21 6481616561296129 q22 489 431 429 429 Total cold run time: 63737 ms Total hot run time: 60383 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](create tablet) default create tablet round robin [doris]
doris-robot commented on PR #28911: URL: https://github.com/apache/doris/pull/28911#issuecomment-1868254861 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit e531f11f36be4fc70cacbca4f957597ffa7ecdcc, data reload: false run tpch-sf100 query with default conf and session variables q1 4714442844784428 q2 369 157 159 157 q3 1464127712201220 q4 1103900 900 900 q5 3222317331943173 q6 250 134 133 133 q7 994 485 490 485 q8 2218220421832183 q9 6689674566956695 q10 3190325133033251 q11 306 194 193 193 q12 355 213 215 213 q13 4599379238253792 q14 243 211 218 211 q15 565 525 524 524 q16 440 391 395 391 q17 1022677 590 590 q18 7080677768466777 q19 1519143714291429 q20 551 327 292 292 q21 3086265326572653 q22 350 277 284 277 Total cold run time: 44329 ms Total hot run time: 39967 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4363432843324328 q2 274 170 175 170 q3 3495348834923488 q4 2405236323722363 q5 5739571657105710 q6 241 123 122 122 q7 2371182918781829 q8 3538352135353521 q9 9036901389418941 q10 3900399840173998 q11 480 356 370 356 q12 771 601 629 601 q13 4289360735573557 q14 283 253 256 253 q15 581 523 526 523 q16 499 472 474 472 q17 1877186618561856 q18 8606813981688139 q19 1746175517921755 q20 2285195819471947 q21 6502612561236123 q22 499 433 417 417 Total cold run time: 63780 ms Total hot run time: 60469 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet) the end offset of column chunk may be wrong in parquet metadata [doris]
doris-robot commented on PR #28893: URL: https://github.com/apache/doris/pull/28893#issuecomment-1868255585 TeamCity be ut coverage result: Function Coverage: 37.74% (7990/21170) Line Coverage: 29.45% (64914/220412) Region Coverage: 28.92% (33391/115464) Branch Coverage: 24.79% (17131/69094) Coverage Report: http://coverage.selectdb-in.cc/coverage/00c92dc619375f720158682affae764c91c269c9_00c92dc619375f720158682affae764c91c269c9/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet) the end offset of column chunk may be wrong in parquet metadata [doris]
doris-robot commented on PR #28891: URL: https://github.com/apache/doris/pull/28891#issuecomment-1868256021 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.25 seconds stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 28.7 seconds inserted 1000 Rows, about 348K ops/s storage size: 17183597790 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](create tablet) default create tablet round robin [doris]
doris-robot commented on PR #28911: URL: https://github.com/apache/doris/pull/28911#issuecomment-1868256348 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.6 seconds stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.6 seconds inserted 1000 Rows, about 349K ops/s storage size: 17183704084 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet) the end offset of column chunk may be wrong in parquet metadata [doris]
doris-robot commented on PR #28893: URL: https://github.com/apache/doris/pull/28893#issuecomment-1868257923 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 51.3 seconds stream load tsv: 574 seconds loaded 74807831229 Bytes, about 124 MB/s stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 30.5 seconds inserted 1000 Rows, about 327K ops/s storage size: 17167903773 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](execute) optimize for ColumnNullable's serialize_vec/deserialize_vec [doris]
BiteThet commented on PR #28788: URL: https://github.com/apache/doris/pull/28788#issuecomment-1868260993 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement] add a lower bound for bytes in scanner queue [doris]
github-actions[bot] commented on PR #28912: URL: https://github.com/apache/doris/pull/28912#issuecomment-1868266539 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](execute) optimize for ColumnNullable's serialize_vec/deserialize_vec [doris]
doris-robot commented on PR #28788: URL: https://github.com/apache/doris/pull/28788#issuecomment-1868267018 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit 2a83c1f25a87f09e5a9cbedca5d51875c9baa2de, data reload: false run tpch-sf100 query with default conf and session variables q1 4745443344584433 q2 373 144 157 144 q3 1463124412271227 q4 908 919 908 q5 3160317731783177 q6 249 136 133 133 q7 1011495 492 492 q8 2184221622022202 q9 6690668966636663 q10 3243326832883268 q11 309 179 183 179 q12 352 216 212 212 q13 4565382638263826 q14 242 215 216 215 q15 573 521 533 521 q16 438 389 383 383 q17 1028693 533 533 q18 7013674468736744 q19 1523140014261400 q20 509 292 309 292 q21 3064265626932656 q22 355 280 290 280 Total cold run time: 44200 ms Total hot run time: 39888 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4391433743444337 q2 272 164 180 164 q3 3518349934963496 q4 2402237723782377 q5 5714570457075704 q6 240 121 123 121 q7 2366185518931855 q8 3532352835193519 q9 9080902490149014 q10 3904401940274019 q11 483 371 367 367 q12 771 592 611 592 q13 4275357535513551 q14 296 251 257 251 q15 577 529 527 527 q16 512 454 462 454 q17 1895183918651839 q18 8559814180208020 q19 1774175217271727 q20 2256196119241924 q21 6544615761156115 q22 505 441 446 441 Total cold run time: 63866 ms Total hot run time: 60414 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](chore) update dcheck to avoid core during stress test (#28895)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new de6c7a792e9 [fix](chore) update dcheck to avoid core during stress test (#28895) de6c7a792e9 is described below commit de6c7a792e95ea8d12b0f20677d1ee567bbec81a Author: zhannngchen <48427519+zhannngc...@users.noreply.github.com> AuthorDate: Sat Dec 23 18:49:57 2023 +0800 [fix](chore) update dcheck to avoid core during stress test (#28895) --- be/src/olap/tablet.cpp | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp index 47fd726b375..a5f34633d79 100644 --- a/be/src/olap/tablet.cpp +++ b/be/src/olap/tablet.cpp @@ -3016,7 +3016,10 @@ Status Tablet::calc_segment_delete_bitmap(RowsetSharedPtr rowset, auto st = lookup_row_key(key, true, specified_rowsets, &loc, dummy_version.first - 1, segment_caches, &rowset_find); bool expected_st = st.ok() || st.is() || st.is(); -DCHECK(expected_st) << "unexpected error status while lookup_row_key:" << st; +// It's a defensive DCHECK, we need to exclude some common errors to avoid core-dump +// while stress test +DCHECK(expected_st || st.is()) +<< "unexpected error status while lookup_row_key:" << st; if (!expected_st) { return st; } - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](chore) update DCHECK to avoid core during stress test [doris]
dataroaring merged PR #28895: URL: https://github.com/apache/doris/pull/28895 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement] add a lower bound for bytes in scanner queue [doris]
dataroaring commented on PR #28912: URL: https://github.com/apache/doris/pull/28912#issuecomment-1868267755 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement] add a lower bound for bytes in scanner queue [doris]
dataroaring commented on PR #28912: URL: https://github.com/apache/doris/pull/28912#issuecomment-1868267919 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance-wip](multi-catalog) Speed up consume rate of hms events. [doris]
dutyu commented on PR #27666: URL: https://github.com/apache/doris/pull/27666#issuecomment-1868268009 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement] add a lower bound for bytes in scanner queue [doris]
github-actions[bot] commented on PR #28912: URL: https://github.com/apache/doris/pull/28912#issuecomment-1868268502 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](execute) optimize for ColumnNullable's serialize_vec/deserialize_vec [doris]
doris-robot commented on PR #28788: URL: https://github.com/apache/doris/pull/28788#issuecomment-1868269171 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.42 seconds stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.1 seconds inserted 1000 Rows, about 343K ops/s storage size: 17184166728 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement] add a lower bound for bytes in scanner queue [doris]
github-actions[bot] commented on PR #28912: URL: https://github.com/apache/doris/pull/28912#issuecomment-1868269260 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance-wip](multi-catalog) Speed up consume rate of hms events. [doris]
doris-robot commented on PR #27666: URL: https://github.com/apache/doris/pull/27666#issuecomment-1868273738 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit c0c0e137de47d6b3b2ac634161f9027cb4bcba0a, data reload: false run tpch-sf100 query with default conf and session variables q1 4718441044154410 q2 372 143 159 143 q3 1449124612501246 q4 1095907 914 907 q5 3168316531833165 q6 252 138 133 133 q7 1004499 486 486 q8 2175220721972197 q9 6676667366496649 q10 3204328732833283 q11 308 182 189 182 q12 350 210 208 208 q13 4525382937653765 q14 243 218 215 215 q15 568 522 521 521 q16 437 394 391 391 q17 1000653 602 602 q18 7125672168056721 q19 1530144213871387 q20 543 298 286 286 q21 3075264126452641 q22 351 280 281 280 Total cold run time: 44168 ms Total hot run time: 39818 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4347436343434343 q2 277 166 174 166 q3 3521350334963496 q4 2404237123832371 q5 5702570256945694 q6 241 121 121 121 q7 2365190018351835 q8 3517352235183518 q9 9025905990189018 q10 3893399740133997 q11 496 368 367 367 q12 765 589 586 586 q13 4282354935703549 q14 290 255 251 251 q15 579 522 520 520 q16 515 464 462 462 q17 1883184918531849 q18 8444813281598132 q19 1731178217401740 q20 2263195619601956 q21 6482617861306130 q22 519 412 420 412 Total cold run time: 63541 ms Total hot run time: 60513 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement] add a lower bound for bytes in scanner queue [doris]
doris-robot commented on PR #28912: URL: https://github.com/apache/doris/pull/28912#issuecomment-1868275504 TeamCity be ut coverage result: Function Coverage: 36.59% (8549/23364) Line Coverage: 28.66% (69510/242505) Region Coverage: 27.66% (35945/129952) Branch Coverage: 24.40% (18376/75320) Coverage Report: http://coverage.selectdb-in.cc/coverage/bcf21a69ffe78a7c2454e2c7ed078fe27b3de9b9_bcf21a69ffe78a7c2454e2c7ed078fe27b3de9b9/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance-wip](multi-catalog) Speed up consume rate of hms events. [doris]
doris-robot commented on PR #27666: URL: https://github.com/apache/doris/pull/27666#issuecomment-1868275698 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.32 seconds stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.5 seconds inserted 1000 Rows, about 350K ops/s storage size: 17183123141 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement] add a lower bound for bytes in scanner queue [doris]
dataroaring commented on PR #28912: URL: https://github.com/apache/doris/pull/28912#issuecomment-1868277601 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](mtmv)fix can not create mtmv all use default value [doris]
zddr opened a new pull request, #28922: URL: https://github.com/apache/doris/pull/28922 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv)fix can not create mtmv all use default value [doris]
zddr commented on PR #28922: URL: https://github.com/apache/doris/pull/28922#issuecomment-1868278255 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement] add a lower bound for bytes in scanner queue [doris]
github-actions[bot] commented on PR #28912: URL: https://github.com/apache/doris/pull/28912#issuecomment-1868278807 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement] add a lower bound for bytes in scanner queue [doris]
doris-robot commented on PR #28912: URL: https://github.com/apache/doris/pull/28912#issuecomment-1868282857 TeamCity be ut coverage result: Function Coverage: 36.59% (8548/23364) Line Coverage: 28.66% (69502/242505) Region Coverage: 27.66% (35940/129952) Branch Coverage: 24.39% (18371/75320) Coverage Report: http://coverage.selectdb-in.cc/coverage/d06f492f37e2aa3aaffb3dd8fc7abfb745671b48_d06f492f37e2aa3aaffb3dd8fc7abfb745671b48/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv)fix can not create mtmv all use default value [doris]
doris-robot commented on PR #28922: URL: https://github.com/apache/doris/pull/28922#issuecomment-1868284498 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit a7a86d997664870b1af3fdb6b30dc48362912038, data reload: false run tpch-sf100 query with default conf and session variables q1 4683437944484379 q2 369 143 158 143 q3 1465125012881250 q4 1113944 879 879 q5 3159317531713171 q6 248 131 133 131 q7 989 482 490 482 q8 2183219221892189 q9 6678664666726646 q10 3231326132793261 q11 307 179 184 179 q12 363 208 209 208 q13 4545383437493749 q14 245 212 215 212 q15 574 525 519 519 q16 440 384 378 378 q17 1009564 542 542 q18 7000672968256729 q19 1511142414741424 q20 550 331 284 284 q21 3115262726782627 q22 350 275 278 275 Total cold run time: 44127 ms Total hot run time: 39657 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4359434343454343 q2 272 164 177 164 q3 3505349934743474 q4 2380236423622362 q5 5705569156855685 q6 241 123 119 119 q7 2332187319031873 q8 3523352335163516 q9 8983897189048904 q10 3899397140133971 q11 492 378 376 376 q12 756 598 598 598 q13 4285353335603533 q14 288 261 255 255 q15 572 522 521 521 q16 479 444 471 444 q17 1883185618561856 q18 8470813779307930 q19 1775177117541754 q20 2260194819331933 q21 6507612061296120 q22 489 412 432 412 Total cold run time: 63455 ms Total hot run time: 60143 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement] add a lower bound for bytes in scanner queue [doris]
dataroaring commented on PR #28912: URL: https://github.com/apache/doris/pull/28912#issuecomment-1868285980 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv)fix can not create mtmv all use default value [doris]
doris-robot commented on PR #28922: URL: https://github.com/apache/doris/pull/28922#issuecomment-1868286434 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.64 seconds stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17188656333 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Fix query mv rewrite fail when mv cache build quickly [doris]
seawinde commented on PR #28876: URL: https://github.com/apache/doris/pull/28876#issuecomment-1868286552 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](merge-on-write) migration may cause duplicate keys for mow table [doris]
liaoxin01 opened a new pull request, #28923: URL: https://github.com/apache/doris/pull/28923 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](statistics) Fix partition name NPE and sample for all table during auto analyze. [doris]
github-actions[bot] commented on PR #28916: URL: https://github.com/apache/doris/pull/28916#issuecomment-1868286892 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement] add a lower bound for bytes in scanner queue [doris]
github-actions[bot] commented on code in PR #28912: URL: https://github.com/apache/doris/pull/28912#discussion_r1435592850 ## be/src/util/bvar_helper.h: ## @@ -16,6 +16,7 @@ // under the License. #pragma once +#include Review Comment: warning: 'bvar/bvar.h' file not found [clang-diagnostic-error] ```cpp #include ^ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](merge-on-write) fix migration may cause duplicate keys for mow table [doris]
liaoxin01 commented on PR #28923: URL: https://github.com/apache/doris/pull/28923#issuecomment-1868287243 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv)fix can not create mtmv all use default value [doris]
github-actions[bot] commented on PR #28922: URL: https://github.com/apache/doris/pull/28922#issuecomment-1868287407 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv)fix can not create mtmv all use default value [doris]
github-actions[bot] commented on PR #28922: URL: https://github.com/apache/doris/pull/28922#issuecomment-1868287412 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] The documentation of s3 tvf does not include the usage example of BOS [doris]
morningman closed issue #28897: [Bug] The documentation of s3 tvf does not include the usage example of BOS URL: https://github.com/apache/doris/issues/28897 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv)fix can not create mtmv all use default value [doris]
morningman commented on code in PR #28922: URL: https://github.com/apache/doris/pull/28922#discussion_r1435593376 ## fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java: ## @@ -616,6 +616,9 @@ private String getOriginSql(ParserRuleContext ctx) { @Override public MTMVRefreshTriggerInfo visitRefreshTrigger(RefreshTriggerContext ctx) { +if (ctx == null) { Review Comment: If all are default value, better create a default RefreshTriggerContext? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](doc) Add the usage example of bos to the documentation of s3 tvf [doris]
morningman merged PR #28899: URL: https://github.com/apache/doris/pull/28899 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](doc) Add the usage example of bos to the documentation of s3 tvf (#28899)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 5e9e199ca0d [fix](doc) Add the usage example of bos to the documentation of s3 tvf (#28899) 5e9e199ca0d is described below commit 5e9e199ca0deffc1bb5a5923efd63b433cf6a647 Author: nanfeng <42513321+nanfeng1...@users.noreply.github.com> AuthorDate: Sat Dec 23 20:47:50 2023 +0800 [fix](doc) Add the usage example of bos to the documentation of s3 tvf (#28899) --- docs/en/docs/sql-manual/sql-functions/table-functions/s3.md| 10 ++ docs/zh-CN/docs/sql-manual/sql-functions/table-functions/s3.md | 10 ++ 2 files changed, 20 insertions(+) diff --git a/docs/en/docs/sql-manual/sql-functions/table-functions/s3.md b/docs/en/docs/sql-manual/sql-functions/table-functions/s3.md index c482542595d..e410bf39649 100644 --- a/docs/en/docs/sql-manual/sql-functions/table-functions/s3.md +++ b/docs/en/docs/sql-manual/sql-functions/table-functions/s3.md @@ -160,6 +160,16 @@ select * from s3( "region" = "ap-hongkong", "format" = "parquet", "use_path_style" = "false"); + +// The BOS on Baidu Cloud will use 'virtual-hosted style' compatible with the S3 protocol to access s3. +// BOS +select * from s3( +"uri" = "https://example-bucket.s3.bj.bcebos.com/your-folder/file.parquet";, +"s3.access_key"= "ak", +"s3.secret_key" = "sk", +"s3.region" = "bj", +"format" = "parquet", +"use_path_style" = "false"); ``` Example of s3://: diff --git a/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/s3.md b/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/s3.md index 5a9ffd60404..5ee99684a90 100644 --- a/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/s3.md +++ b/docs/zh-CN/docs/sql-manual/sql-functions/table-functions/s3.md @@ -160,6 +160,16 @@ select * from s3( "s3.region" = "ap-hongkong", "format" = "parquet", "use_path_style" = "false"); + +// 百度云bos采用兼容s3协议的virtual-hosted style方式访问s3。 +// BOS +select * from s3( +"uri" = "https://example-bucket.s3.bj.bcebos.com/your-folder/file.parquet";, +"s3.access_key"= "ak", +"s3.secret_key" = "sk", +"s3.region" = "bj", +"format" = "parquet", +"use_path_style" = "false"); ``` s3:// 使用示例: - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](merge-on-write) fix migration may cause duplicate keys for mow table [doris]
github-actions[bot] commented on PR #28923: URL: https://github.com/apache/doris/pull/28923#issuecomment-1868287713 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Fix query mv rewrite fail when mv cache build quickly [doris]
github-actions[bot] commented on PR #28876: URL: https://github.com/apache/doris/pull/28876#issuecomment-1868288037 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Fix query mv rewrite fail when mv cache build quickly [doris]
github-actions[bot] commented on PR #28876: URL: https://github.com/apache/doris/pull/28876#issuecomment-1868288042 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Related partition exclude null generate column when increment build materialized view [doris]
github-actions[bot] commented on PR #28855: URL: https://github.com/apache/doris/pull/28855#issuecomment-1868288266 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Related partition exclude null generate column when increment build materialized view [doris]
github-actions[bot] commented on PR #28855: URL: https://github.com/apache/doris/pull/28855#issuecomment-1868288271 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](statistics)Remove retry load when load stats cache fail. [doris]
github-actions[bot] commented on PR #28904: URL: https://github.com/apache/doris/pull/28904#issuecomment-1868288646 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet) the end offset of column chunk may be wrong in parquet metadata [doris]
github-actions[bot] commented on PR #28891: URL: https://github.com/apache/doris/pull/28891#issuecomment-1868288935 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement] add a lower bound for bytes in scanner queue [doris]
doris-robot commented on PR #28912: URL: https://github.com/apache/doris/pull/28912#issuecomment-1868291291 TeamCity be ut coverage result: Function Coverage: 36.60% (8551/23364) Line Coverage: 28.68% (69548/242505) Region Coverage: 27.67% (35963/129952) Branch Coverage: 24.41% (18383/75320) Coverage Report: http://coverage.selectdb-in.cc/coverage/c7ea8d4a861012c14faed9615c9b77f9eb2582eb_c7ea8d4a861012c14faed9615c9b77f9eb2582eb/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Fix query mv rewrite fail when mv cache build quickly [doris]
doris-robot commented on PR #28876: URL: https://github.com/apache/doris/pull/28876#issuecomment-1868292351 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit ccd458dfbc6ba05f57e34b86e37683e0a5f98ace, data reload: false run tpch-sf100 query with default conf and session variables q1 4678440844504408 q2 371 141 161 141 q3 1470125212511251 q4 1127907 885 885 q5 3212316031513151 q6 245 130 130 130 q7 976 497 483 483 q8 2188222021912191 q9 6710667366496649 q10 3239328032743274 q11 308 193 181 181 q12 354 211 204 204 q13 4546379538023795 q14 249 220 208 208 q15 572 524 516 516 q16 440 383 390 383 q17 1019576 529 529 q18 7040675568446755 q19 1523143014541430 q20 537 297 313 297 q21 3087263427012634 q22 345 276 277 276 Total cold run time: 44236 ms Total hot run time: 39771 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4347434543394339 q2 268 163 174 163 q3 3502349834853485 q4 2398237723782377 q5 5699570757235707 q6 240 124 122 122 q7 2345184418771844 q8 3507352435233523 q9 8998896289798962 q10 3917400639953995 q11 475 374 360 360 q12 770 599 589 589 q13 4303357435353535 q14 300 242 258 242 q15 574 519 520 519 q16 511 486 473 473 q17 1876186118511851 q18 8549815780908090 q19 1746177717691769 q20 2249196419291929 q21 6488618261556155 q22 501 423 429 423 Total cold run time: 63563 ms Total hot run time: 60452 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv)fix can not create mtmv all use default value [doris]
morningman commented on code in PR #28922: URL: https://github.com/apache/doris/pull/28922#discussion_r1435599823 ## fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java: ## @@ -616,6 +616,9 @@ private String getOriginSql(ParserRuleContext ctx) { @Override public MTMVRefreshTriggerInfo visitRefreshTrigger(RefreshTriggerContext ctx) { +if (ctx == null) { Review Comment: I got it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](merge-on-write) fix migration may cause duplicate keys for mow table [doris]
doris-robot commented on PR #28923: URL: https://github.com/apache/doris/pull/28923#issuecomment-1868293238 TeamCity be ut coverage result: Function Coverage: 36.59% (8550/23364) Line Coverage: 28.67% (69517/242500) Region Coverage: 27.66% (35941/129945) Branch Coverage: 24.40% (18376/75326) Coverage Report: http://coverage.selectdb-in.cc/coverage/3f165e583563f3611dd739f033c98c21276a8661_3f165e583563f3611dd739f033c98c21276a8661/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](merge-on-write) fix migration may cause duplicate keys for mow table [doris]
doris-robot commented on PR #28923: URL: https://github.com/apache/doris/pull/28923#issuecomment-1868293974 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit 3f165e583563f3611dd739f033c98c21276a8661, data reload: false run tpch-sf100 query with default conf and session variables q1 4686441044354410 q2 369 146 159 146 q3 1472125212191219 q4 1113905 903 903 q5 3163315332043153 q6 254 133 138 133 q7 999 484 495 484 q8 2198222021872187 q9 6700664166986641 q10 3216330932673267 q11 310 188 194 188 q12 356 213 210 210 q13 4593383338123812 q14 240 215 214 214 q15 583 520 520 520 q16 445 384 385 384 q17 1017717 563 563 q18 7160689467996799 q19 1521146214351435 q20 508 323 309 309 q21 3118269726742674 q22 350 281 290 281 Total cold run time: 44371 ms Total hot run time: 39932 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4346434243374337 q2 272 170 180 170 q3 3508348334983483 q4 2415238423772377 q5 5711573557225722 q6 243 123 126 123 q7 2375186618381838 q8 3528353935313531 q9 9034900689988998 q10 3887397440113974 q11 486 360 373 360 q12 779 600 594 594 q13 4299357235573557 q14 286 248 262 248 q15 580 521 512 512 q16 506 479 451 451 q17 1886187818831878 q18 8563811881138113 q19 1752175417441744 q20 2251193119261926 q21 6512616961656165 q22 521 433 423 423 Total cold run time: 63740 ms Total hot run time: 60524 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](statistics) Fix partition name NPE and sample for all table during auto analyze. [doris]
Jibing-Li commented on PR #28916: URL: https://github.com/apache/doris/pull/28916#issuecomment-1868294074 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Fix query mv rewrite fail when mv cache build quickly [doris]
doris-robot commented on PR #28876: URL: https://github.com/apache/doris/pull/28876#issuecomment-1868294520 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.29 seconds stream load tsv: 570 seconds loaded 74807831229 Bytes, about 125 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 28.7 seconds inserted 1000 Rows, about 348K ops/s storage size: 17184424703 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](mtmv)fix can not create mtmv all use default value [doris]
morningman merged PR #28922: URL: https://github.com/apache/doris/pull/28922 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](mtmv)fix can not create mtmv all use default value (#28922)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 66b14f4db1f [fix](mtmv)fix can not create mtmv all use default value (#28922) 66b14f4db1f is described below commit 66b14f4db1f34371237f45b9bca64fc9ef951cc4 Author: zhangdong <493738...@qq.com> AuthorDate: Sat Dec 23 21:27:01 2023 +0800 [fix](mtmv)fix can not create mtmv all use default value (#28922) --- .../org/apache/doris/nereids/parser/LogicalPlanBuilder.java | 6 ++ regression-test/suites/mtmv_p0/test_build_mtmv.groovy | 13 + 2 files changed, 19 insertions(+) diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java index dcbfb9ef3f0..37b42c7123f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java @@ -616,6 +616,9 @@ public class LogicalPlanBuilder extends DorisParserBaseVisitor { @Override public MTMVRefreshTriggerInfo visitRefreshTrigger(RefreshTriggerContext ctx) { +if (ctx == null) { +return new MTMVRefreshTriggerInfo(RefreshTrigger.MANUAL); +} if (ctx.MANUAL() != null) { return new MTMVRefreshTriggerInfo(RefreshTrigger.MANUAL); } @@ -662,6 +665,9 @@ public class LogicalPlanBuilder extends DorisParserBaseVisitor { @Override public BuildMode visitBuildMode(BuildModeContext ctx) { +if (ctx == null) { +return BuildMode.IMMEDIATE; +} if (ctx.DEFERRED() != null) { return BuildMode.DEFERRED; } else if (ctx.IMMEDIATE() != null) { diff --git a/regression-test/suites/mtmv_p0/test_build_mtmv.groovy b/regression-test/suites/mtmv_p0/test_build_mtmv.groovy index 882a7eff22e..eb4560f8b06 100644 --- a/regression-test/suites/mtmv_p0/test_build_mtmv.groovy +++ b/regression-test/suites/mtmv_p0/test_build_mtmv.groovy @@ -152,6 +152,19 @@ suite("test_build_mtmv") { DROP MATERIALIZED VIEW ${mvName} """ +// use default value +sql """ +CREATE MATERIALIZED VIEW ${mvName} +DISTRIBUTED BY RANDOM BUCKETS 2 +PROPERTIES ('replication_num' = '1') +AS +SELECT ${tableName}.username, ${tableNamePv}.pv FROM ${tableName}, ${tableNamePv} WHERE ${tableName}.id=${tableNamePv}.id; +""" + +sql """ +DROP MATERIALIZED VIEW ${mvName} +""" + // IMMEDIATE schedule interval sql """ CREATE MATERIALIZED VIEW ${mvName} - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
Yukang-Lian commented on PR #27726: URL: https://github.com/apache/doris/pull/27726#issuecomment-1868294892 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](statistics) Fix partition name NPE and sample for all table during auto analyze. [doris]
github-actions[bot] commented on PR #28916: URL: https://github.com/apache/doris/pull/28916#issuecomment-1868295154 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](merge-on-write) fix migration may cause duplicate keys for mow table [doris]
doris-robot commented on PR #28923: URL: https://github.com/apache/doris/pull/28923#issuecomment-1868296156 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.07 seconds stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.8 seconds inserted 1000 Rows, about 347K ops/s storage size: 17184037176 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.0-var](disk balance) Impr disk rebalancer sched #26412 [doris]
yujun777 commented on PR #28920: URL: https://github.com/apache/doris/pull/28920#issuecomment-1868296612 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Fix query mv rewrite fail when mv cache build quickly [doris]
seawinde commented on PR #28876: URL: https://github.com/apache/doris/pull/28876#issuecomment-1868297143 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](statistics) Fix partition name NPE and sample for all table during auto analyze. [doris]
doris-robot commented on PR #28916: URL: https://github.com/apache/doris/pull/28916#issuecomment-1868297805 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit 389c80c8461afd01ca28ea07b375c076a404b855, data reload: false run tpch-sf100 query with default conf and session variables q1 4724443944804439 q2 378 185 167 167 q3 1467127112491249 q4 1124907 920 907 q5 3182314631643146 q6 248 129 130 129 q7 994 487 493 487 q8 220621742174 q9 6688662866596628 q10 3220328032443244 q11 312 194 196 194 q12 358 218 212 212 q13 4564383138013801 q14 243 210 213 210 q15 574 528 528 528 q16 439 387 388 387 q17 1005559 509 509 q18 7059675867836758 q19 1511144014551440 q20 537 300 337 300 q21 3087265326672653 q22 351 278 287 278 Total cold run time: 44271 ms Total hot run time: 39840 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4389437643874376 q2 268 165 173 165 q3 3502349835083498 q4 2397237123662366 q5 5715571957185718 q6 240 126 124 124 q7 2360183718861837 q8 3525352235063506 q9 9087900489638963 q10 3915397040053970 q11 487 357 375 357 q12 769 607 606 606 q13 4289354035613540 q14 288 250 255 250 q15 576 530 526 526 q16 503 455 455 455 q17 1888184718341834 q18 8438806780138013 q19 1754177417501750 q20 2241195719201920 q21 6494614160976097 q22 514 423 444 423 Total cold run time: 63639 ms Total hot run time: 60294 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
doris-robot commented on PR #27726: URL: https://github.com/apache/doris/pull/27726#issuecomment-1868299348 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit 5ee0e280a4c6dbb488d8b1da7221546ee51903e7, data reload: false run tpch-sf100 query with default conf and session variables q1 4687445344434443 q2 387 138 157 138 q3 1458127812221222 q4 1105884 884 884 q5 3178320832023202 q6 252 130 136 130 q7 1009486 496 486 q8 2194223522152215 q9 6700669566576657 q10 3205328232843282 q11 306 192 185 185 q12 358 204 212 204 q13 4570379937793779 q14 238 213 223 213 q15 574 524 514 514 q16 442 389 386 386 q17 1017696 581 581 q18 7046674869786748 q19 1535142014791420 q20 531 312 310 310 q21 3101267026442644 q22 345 282 281 281 Total cold run time: 44238 ms Total hot run time: 39924 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4360439143864386 q2 274 165 172 165 q3 3529349835043498 q4 2408237523802375 q5 5731569857215698 q6 246 123 124 123 q7 2372183618261826 q8 3533354535433543 q9 9081900690299006 q10 3920398639953986 q11 488 368 371 368 q12 772 592 611 592 q13 4279354135353535 q14 290 252 254 252 q15 567 522 520 520 q16 499 478 467 467 q17 1909189318771877 q18 8570822181408140 q19 1761176917701769 q20 2250193919431939 q21 6518620461516151 q22 494 431 416 416 Total cold run time: 63851 ms Total hot run time: 60632 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
doris-robot commented on PR #27726: URL: https://github.com/apache/doris/pull/27726#issuecomment-1868299849 TeamCity be ut coverage result: Function Coverage: 36.61% (8560/23383) Line Coverage: 28.68% (69623/242747) Region Coverage: 27.66% (35996/130121) Branch Coverage: 24.39% (18399/75448) Coverage Report: http://coverage.selectdb-in.cc/coverage/5ee0e280a4c6dbb488d8b1da7221546ee51903e7_5ee0e280a4c6dbb488d8b1da7221546ee51903e7/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](statistics) Fix partition name NPE and sample for all table during auto analyze. [doris]
doris-robot commented on PR #28916: URL: https://github.com/apache/doris/pull/28916#issuecomment-1868299805 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.77 seconds stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 29.0 seconds inserted 1000 Rows, about 344K ops/s storage size: 17184573743 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Fix query mv rewrite fail when mv cache build quickly [doris]
doris-robot commented on PR #28876: URL: https://github.com/apache/doris/pull/28876#issuecomment-1868301024 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit a98e4398046eed969d936669e007bcfff403bdc8, data reload: false run tpch-sf100 query with default conf and session variables q1 4692443044474430 q2 378 138 158 138 q3 1444121111941194 q4 1112890 919 890 q5 3151314432043144 q6 245 132 129 129 q7 1028515 491 491 q8 2190223821882188 q9 6698667866476647 q10 3220328032713271 q11 310 181 183 181 q12 354 211 210 210 q13 4545382437983798 q14 246 211 211 211 q15 568 523 520 520 q16 444 387 394 387 q17 1021614 565 565 q18 7082681767436743 q19 1532140113681368 q20 543 311 284 284 q21 3083263026402630 q22 346 285 278 278 Total cold run time: 44232 ms Total hot run time: 39697 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4326434643444344 q2 269 166 169 166 q3 3517349534843484 q4 2404237323792373 q5 5687569657715696 q6 240 122 125 122 q7 2385183818771838 q8 3535353335263526 q9 9024898689768976 q10 3914399439803980 q11 489 386 375 375 q12 760 603 591 591 q13 4302356035543554 q14 291 251 263 251 q15 575 517 518 517 q16 494 468 448 448 q17 1895183418361834 q18 8560818881378137 q19 1738175317361736 q20 2257194519271927 q21 6502617661086108 q22 503 413 429 413 Total cold run time: 63667 ms Total hot run time: 60396 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhancement](Wal)Support dynamic wal space limit [doris]
doris-robot commented on PR #27726: URL: https://github.com/apache/doris/pull/27726#issuecomment-1868301669 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.19 seconds stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.3 seconds inserted 1000 Rows, about 353K ops/s storage size: 17183750098 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement] add a lower bound for bytes in scanner queue [doris]
doris-robot commented on PR #28912: URL: https://github.com/apache/doris/pull/28912#issuecomment-1868302410 TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Tpch sf100 test result on commit c7ea8d4a861012c14faed9615c9b77f9eb2582eb, data reload: false run tpch-sf100 query with default conf and session variables q1 4760432144714321 q2 374 178 158 158 q3 1425118212071182 q4 1073862 760 760 q5 3215312131853121 q6 228 130 136 130 q7 948 466 474 466 q8 2145221321772177 q9 6608657165556555 q10 3192311530773077 q11 297 180 182 180 q12 358 205 205 205 q13 4533376637713766 q14 234 211 211 211 q15 550 511 508 508 q16 432 405 409 405 q17 995 682 529 529 q18 6390603063826030 q19 1560137414371374 q20 521 335 307 307 q21 2935250524652465 q22 346 274 284 274 Total cold run time: 43119 ms Total hot run time: 38201 ms run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off q1 4323435342174217 q2 303 228 230 228 q3 3224299530182995 q4 2100190218891889 q5 5291525052535250 q6 240 122 122 122 q7 2209177718071777 q8 3317342634043404 q9 8584856584688468 q10 3866376537853765 q11 545 433 422 422 q12 743 633 581 581 q13 4319359635213521 q14 286 266 264 264 q15 560 513 509 509 q16 524 501 486 486 q17 1852166616741666 q18 7758740480027404 q19 1785172917141714 q20 2257198119711971 q21 5179517250025002 q22 550 431 439 431 Total cold run time: 59815 ms Total hot run time: 56086 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids) Fix query mv rewrite fail when mv cache build quickly [doris]
doris-robot commented on PR #28876: URL: https://github.com/apache/doris/pull/28876#issuecomment-1868303534 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.55 seconds stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17183822725 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet) the end offset of column chunk may be wrong in parquet metadata [doris]
morningman merged PR #28891: URL: https://github.com/apache/doris/pull/28891 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](parquet) the end offset of column chunk may be wrong in parquet metadata (#28891)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 96d4778f2ec [fix](parquet) the end offset of column chunk may be wrong in parquet metadata (#28891) 96d4778f2ec is described below commit 96d4778f2ec837eef5bfe94b18bde4ccd0400c23 Author: Ashin Gau AuthorDate: Sat Dec 23 22:21:04 2023 +0800 [fix](parquet) the end offset of column chunk may be wrong in parquet metadata (#28891) --- .../vec/exec/format/parquet/vparquet_column_chunk_reader.cpp | 2 ++ .../vec/exec/format/parquet/vparquet_column_chunk_reader.h | 11 ++- be/src/vec/exec/format/parquet/vparquet_page_reader.h| 5 + regression-test/data/external_table_p2/tvf/test_tvf_p2.out | 12 .../suites/external_table_p2/tvf/test_tvf_p2.groovy | 6 ++ 5 files changed, 35 insertions(+), 1 deletion(-) diff --git a/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp b/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp index 928b5ae70bd..6feb9bc1025 100644 --- a/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp +++ b/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp @@ -97,10 +97,12 @@ Status ColumnChunkReader::next_page() { return next_page(); } else if (_page_reader->get_page_header()->type == tparquet::PageType::DATA_PAGE_V2) { _remaining_num_values = _page_reader->get_page_header()->data_page_header_v2.num_values; +_chunk_parsed_values += _remaining_num_values; _state = HEADER_PARSED; return Status::OK(); } else { _remaining_num_values = _page_reader->get_page_header()->data_page_header.num_values; +_chunk_parsed_values += _remaining_num_values; _state = HEADER_PARSED; return Status::OK(); } diff --git a/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.h b/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.h index 21ee808e48f..c8a49e098a5 100644 --- a/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.h +++ b/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.h @@ -91,9 +91,17 @@ public: Status init(); // Whether the chunk reader has a more page to read. -bool has_next_page() { return _page_reader->has_next_page(); } +bool has_next_page() { return _chunk_parsed_values < _metadata.num_values; } +// Deprecated // Seek to the specific page, page_header_offset must be the start offset of the page header. +// _end_offset may exceed the actual data area, so we can only use the number of parsed values +// to determine whether there are remaining pages to read. That's to say we can't use the +// PageLocation in parquet metadata to seek to the specified page. We should call next_page() +// and skip_page() to skip pages one by one. +// todo: change this interface to seek_to_page(int64_t page_header_offset, size_t num_parsed_values) +// and set _chunk_parsed_values = num_parsed_values +// [[deprecated]] void seek_to_page(int64_t page_header_offset) { _remaining_num_values = 0; _page_reader->seek_to_page(page_header_offset); @@ -201,6 +209,7 @@ private: LevelDecoder _rep_level_decoder; LevelDecoder _def_level_decoder; +size_t _chunk_parsed_values = 0; uint32_t _remaining_num_values = 0; Slice _page_data; std::unique_ptr _decompress_buf; diff --git a/be/src/vec/exec/format/parquet/vparquet_page_reader.h b/be/src/vec/exec/format/parquet/vparquet_page_reader.h index 730b9a3001b..bdd0a8d0f5f 100644 --- a/be/src/vec/exec/format/parquet/vparquet_page_reader.h +++ b/be/src/vec/exec/format/parquet/vparquet_page_reader.h @@ -45,6 +45,11 @@ public: uint64_t length); ~PageReader() = default; +// Deprecated +// Parquet file may not be standardized, +// _end_offset may exceed the actual data area. +// ColumnChunkReader::has_next_page() use the number of parsed values for judgment +// [[deprecated]] bool has_next_page() const { return _offset < _end_offset; } Status next_page_header(); diff --git a/regression-test/data/external_table_p2/tvf/test_tvf_p2.out b/regression-test/data/external_table_p2/tvf/test_tvf_p2.out index 6a44b7322dc..53b454df858 100644 --- a/regression-test/data/external_table_p2/tvf/test_tvf_p2.out +++ b/regression-test/data/external_table_p2/tvf/test_tvf_p2.out @@ -50,6 +50,18 @@ 32.024 64. 128.901468 32.024 64. 128.901468 2023-07-07 2023-07-07 2021-07-07T19:15:31.123456 2023-07-07 2023-07-07 2021-07-07T19:15:31.123456 32.689 64.2580 128.745382 32.689 64.2580 128.745382 2023-11-11 2023-11-11 2022-11-11T16:35:37.123456 2023-11-11 2023-11-11 2022-11-11T16:35:37.1
Re: [PR] [fix](parquet) the end offset of column chunk may be wrong in parquet metadata [doris]
github-actions[bot] commented on PR #28891: URL: https://github.com/apache/doris/pull/28891#issuecomment-1868304185 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](parquet) the end offset of column chunk may be wrong in parquet metadata [doris]
morningman merged PR #28893: URL: https://github.com/apache/doris/pull/28893 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [fix](parquet) the end offset of column chunk may be wrong in parquet metadata #28891 (#28893)
This is an automated email from the ASF dual-hosted git repository. morningman pushed a commit to branch branch-2.0 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.0 by this push: new adb7730b64b [fix](parquet) the end offset of column chunk may be wrong in parquet metadata #28891 (#28893) adb7730b64b is described below commit adb7730b64b0a0d882fec6517af9c2dc73cdb7b5 Author: Ashin Gau AuthorDate: Sat Dec 23 22:23:07 2023 +0800 [fix](parquet) the end offset of column chunk may be wrong in parquet metadata #28891 (#28893) backport: #28891 --- .../vec/exec/format/parquet/vparquet_column_chunk_reader.cpp | 2 ++ .../vec/exec/format/parquet/vparquet_column_chunk_reader.h | 11 ++- be/src/vec/exec/format/parquet/vparquet_page_reader.h| 5 + regression-test/data/external_table_p2/tvf/test_tvf_p2.out | 12 .../suites/external_table_p2/tvf/test_tvf_p2.groovy | 6 ++ 5 files changed, 35 insertions(+), 1 deletion(-) diff --git a/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp b/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp index aca5ae96e81..579bfa5a4ae 100644 --- a/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp +++ b/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.cpp @@ -98,10 +98,12 @@ Status ColumnChunkReader::next_page() { return next_page(); } else if (_page_reader->get_page_header()->type == tparquet::PageType::DATA_PAGE_V2) { _remaining_num_values = _page_reader->get_page_header()->data_page_header_v2.num_values; +_chunk_parsed_values += _remaining_num_values; _state = HEADER_PARSED; return Status::OK(); } else { _remaining_num_values = _page_reader->get_page_header()->data_page_header.num_values; +_chunk_parsed_values += _remaining_num_values; _state = HEADER_PARSED; return Status::OK(); } diff --git a/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.h b/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.h index 9f62fddce70..aa30af1e94c 100644 --- a/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.h +++ b/be/src/vec/exec/format/parquet/vparquet_column_chunk_reader.h @@ -91,9 +91,17 @@ public: Status init(); // Whether the chunk reader has a more page to read. -bool has_next_page() { return _page_reader->has_next_page(); } +bool has_next_page() { return _chunk_parsed_values < _metadata.num_values; } +// Deprecated // Seek to the specific page, page_header_offset must be the start offset of the page header. +// _end_offset may exceed the actual data area, so we can only use the number of parsed values +// to determine whether there are remaining pages to read. That's to say we can't use the +// PageLocation in parquet metadata to seek to the specified page. We should call next_page() +// and skip_page() to skip pages one by one. +// todo: change this interface to seek_to_page(int64_t page_header_offset, size_t num_parsed_values) +// and set _chunk_parsed_values = num_parsed_values +// [[deprecated]] void seek_to_page(int64_t page_header_offset) { _remaining_num_values = 0; _page_reader->seek_to_page(page_header_offset); @@ -201,6 +209,7 @@ private: LevelDecoder _rep_level_decoder; LevelDecoder _def_level_decoder; +size_t _chunk_parsed_values = 0; uint32_t _remaining_num_values = 0; Slice _page_data; std::unique_ptr _decompress_buf; diff --git a/be/src/vec/exec/format/parquet/vparquet_page_reader.h b/be/src/vec/exec/format/parquet/vparquet_page_reader.h index 089409786f4..3b816629c6e 100644 --- a/be/src/vec/exec/format/parquet/vparquet_page_reader.h +++ b/be/src/vec/exec/format/parquet/vparquet_page_reader.h @@ -45,6 +45,11 @@ public: uint64_t length); ~PageReader() = default; +// Deprecated +// Parquet file may not be standardized, +// _end_offset may exceed the actual data area. +// ColumnChunkReader::has_next_page() use the number of parsed values for judgment +// [[deprecated]] bool has_next_page() const { return _offset < _end_offset; } Status next_page_header(); diff --git a/regression-test/data/external_table_p2/tvf/test_tvf_p2.out b/regression-test/data/external_table_p2/tvf/test_tvf_p2.out index 6a44b7322dc..53b454df858 100644 --- a/regression-test/data/external_table_p2/tvf/test_tvf_p2.out +++ b/regression-test/data/external_table_p2/tvf/test_tvf_p2.out @@ -50,6 +50,18 @@ 32.024 64. 128.901468 32.024 64. 128.901468 2023-07-07 2023-07-07 2021-07-07T19:15:31.123456 2023-07-07 2023-07-07 2021-07-07T19:15:31.123456 32.689 64.2580 128.745382 32.689 64.2580 128.745382 2023-11-11 2023-11-11 2022-11-11T16:35:37.123456 2023-1
Re: [PR] [improvement] add a lower bound for bytes in scanner queue [doris]
doris-robot commented on PR #28912: URL: https://github.com/apache/doris/pull/28912#issuecomment-1868304670 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 44.68 seconds stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 28.9 seconds inserted 1000 Rows, about 346K ops/s storage size: 17183644549 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org