Re: [PR] [fix](planner) remove input slot for aggregate slot which is not materialized [doris]
doris-robot commented on PR #32092: URL: https://github.com/apache/doris/pull/32092#issuecomment-1990909281 TPC-DS: Total hot run time: 187409 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit c325499b1b02c504cf6414c4b1d94da1b0eaf894, data reload: false query1 918 363 342 342 query2 7520199719651965 query3 6699212 213 212 query4 31178 20603 20614 20603 query5 4399476 406 406 query6 277 179 174 174 query7 4623320 290 290 query8 227 177 168 168 query9 9965232223212321 query10 450 242 247 242 query11 14617 14170 14136 14136 query12 130 91 83 83 query13 1648411 432 411 query14 11238 10209 10588 10209 query15 252 207 194 194 query16 7842263 252 252 query17 1962556 523 523 query18 1973276 279 276 query19 214 160 142 142 query20 87 83 82 82 query21 198 131 121 121 query22 4837454145994541 query23 31660 31063 31013 31013 query24 11068 282728322827 query25 629 356 407 356 query26 1575152 158 152 query27 3139361 356 356 query28 8241192218761876 query29 1002640 624 624 query30 306 148 150 148 query31 950 690 729 690 query32 91 59 53 53 query33 767 261 252 252 query34 1063473 474 473 query35 846 628 603 603 query36 983 884 863 863 query37 122 75 76 75 query38 3336319132043191 query39 1421141113801380 query40 233 112 111 111 query41 51 47 48 47 query42 96 97 96 96 query43 490 447 461 447 query44 1261711 719 711 query45 273 227 256 227 query46 1089706 693 693 query47 1676160716011601 query48 455 360 363 360 query49 1165332 347 332 query50 754 375 374 374 query51 6757662666866626 query52 105 91 95 91 query53 341 267 268 267 query54 326 249 248 248 query55 85 81 80 80 query56 245 234 233 233 query57 1084100510281005 query58 236 199 208 199 query59 2774264426342634 query60 263 256 251 251 query61 113 112 110 110 query62 589 392 404 392 query63 304 325 276 276 query64 6935381939363819 query65 3095302930533029 query66 1402331 324 324 query67 15119 14840 14582 14582 query68 9231551 535 535 query69 666 386 380 380 query70 1399120712011201 query71 521 278 269 269 query72 6585264924872487 query73 1622328 328 328 query74 7677658566026585 query75 9723799680047996 query76 5615949 977 949 query77 634 258 263 258 query78 10444 979497239723 query79 11625 523 515 515 query80 2241392 379 379 query81 485 206 207 206 query82 333 203 197 197 query83 244 141 147 141 query84 280 73 80 73 query85 1240324 319 319 query86 351 277 308 277 query87 3543330632693269 query88 5283235123652351 query89 463 357 352 352 query90 2058174 171 171 query91 157 129 131 129 query92 57 47 46 46 query93 6539515 499 499 query94 1375173 174 173 query95 415 325 320 320 query96 606 269 270 269 query97 4429425842914258 query98 224 209 204 204 query99 973 760 762 760 Total cold run time: 322807 ms Total hot run time: 187409 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t
Re: [PR] [fix](planner) remove input slot for aggregate slot which is not materialized [doris]
doris-robot commented on PR #32092: URL: https://github.com/apache/doris/pull/32092#issuecomment-1990914891 ClickBench: Total hot run time: 30.44 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit c325499b1b02c504cf6414c4b1d94da1b0eaf894, data reload: false query1 0.040.030.03 query2 0.050.020.02 query3 0.230.060.06 query4 1.680.100.09 query5 0.480.520.51 query6 1.210.620.62 query7 0.020.010.01 query8 0.040.030.03 query9 0.540.500.50 query10 0.540.550.55 query11 0.110.090.09 query12 0.110.090.09 query13 0.570.570.56 query14 0.740.740.76 query15 0.820.780.79 query16 0.380.400.39 query17 0.980.951.00 query18 0.180.280.22 query19 1.731.661.68 query20 0.010.000.01 query21 15.41 0.560.56 query22 2.692.392.06 query23 17.18 0.760.78 query24 3.542.670.86 query25 0.100.120.16 query26 0.750.120.14 query27 0.030.030.03 query28 10.17 0.930.85 query29 12.53 3.193.19 query30 0.680.520.59 query31 2.800.360.34 query32 3.440.440.45 query33 2.792.802.82 query34 16.00 4.344.31 query35 4.314.304.37 query36 1.081.000.99 query37 0.070.050.06 query38 0.040.030.03 query39 0.020.020.02 query40 0.170.130.14 query41 0.080.020.02 query42 0.030.020.02 query43 0.030.020.02 Total cold run time: 104.4 s Total hot run time: 30.44 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Fix LRU cache deleter and memory tracking [doris]
xinyiZzz commented on code in PR #32080: URL: https://github.com/apache/doris/pull/32080#discussion_r1520949552 ## be/src/runtime/memory/lru_cache_policy.h: ## @@ -72,7 +81,48 @@ class LRUCachePolicy : public CachePolicy { return true; } -~LRUCachePolicy() override = default; +static std::string lru_cache_type_string(LRUCacheType type) { +switch (type) { +case LRUCacheType::SIZE: +return "size"; +case LRUCacheType::NUMBER: +return "number"; +default: +LOG(FATAL) << "not match type of lru cache:" << static_cast(type); +} +} + +std::shared_ptr mem_tracker() { return _mem_tracker; } + +Cache::Handle* insert(const CacheKey& key, void* value, size_t charge, + CachePriority priority = CachePriority::NORMAL, size_t bytes = -1) { +return _insert(true, key, value, charge, priority, bytes); +} + +Cache::Handle* insert_no_tracking(const CacheKey& key, void* value, size_t charge, Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Fix LRU cache deleter and memory tracking [doris]
xinyiZzz commented on PR #32080: URL: https://github.com/apache/doris/pull/32080#issuecomment-1990915459 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] (enhance)(Cache) Control whether writing into file cache for query and load [doris]
ByteYue opened a new pull request, #32099: URL: https://github.com/apache/doris/pull/32099 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (enhance)(Cache) Control whether writing into file cache for query and load [doris]
doris-robot commented on PR #32099: URL: https://github.com/apache/doris/pull/32099#issuecomment-1990918211 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](routine-load)add rack tag into fe & kafka routine load(#31531) [doris]
HHoflittlefish777 commented on code in PR #32098: URL: https://github.com/apache/doris/pull/32098#discussion_r1520951488 ## fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java: ## @@ -93,6 +95,7 @@ public class KafkaRoutineLoadJob extends RoutineLoadJob { // kafka properties ,property prefix will be mapped to kafka custom parameters, which can be extended in the future private Map customProperties = Maps.newHashMap(); private Map convertedCustomProperties = Maps.newHashMap(); +private Map> convertedRackCustomProperties = Maps.newHashMap(); Review Comment: why do you use a new property "convertedRackCustomProperties" to store rack info rather than in convertedCustomProperties? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](lookup)compatible with lookup timestamp types [doris-flink-connector]
vinlee19 opened a new pull request, #341: URL: https://github.com/apache/doris-flink-connector/pull/341 # Proposed changes Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 2. Has unit tests been added: (Yes/No/No Need) 3. Has document been added or modified: (Yes/No/No Need) 4. Does it need to update dependencies: (Yes/No) 5. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](planner) remove input slot for aggregate slot which is not materialized [doris]
doris-robot commented on PR #32092: URL: https://github.com/apache/doris/pull/32092#issuecomment-1990918396 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit c325499b1b02c504cf6414c4b1d94da1b0eaf894 with default session variables Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 17.9 seconds inserted 1000 Rows, about 558K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](Nereids) Optimize expression [doris]
924060929 commented on PR #32067: URL: https://github.com/apache/doris/pull/32067#issuecomment-1990918902 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Fix](trino-connector) Fix the bug of query wrongs when using trino-connector [doris]
BePPPower opened a new pull request, #32100: URL: https://github.com/apache/doris/pull/32100 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](trino-connector) Fix the bug of query wrongs when using trino-connector [doris]
doris-robot commented on PR #32100: URL: https://github.com/apache/doris/pull/32100#issuecomment-1990922257 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](routine-load)add rack tag into fe & kafka routine load(#31531) [doris]
HHoflittlefish777 commented on PR #32098: URL: https://github.com/apache/doris/pull/32098#issuecomment-1990922774 Is rack-aware work when use routine load after this pr? Can you add more test to verify it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](Cloud) Add property storage vault for OlapTable [doris]
doris-robot commented on PR #31628: URL: https://github.com/apache/doris/pull/31628#issuecomment-1990923837 TPC-H: Total hot run time: 36920 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 43c0b6563d19d69d4fb769f3a5bc168dbd33ef6c, data reload: false -- Round 1 -- q1 17629 428141394139 q2 2021156 143 143 q3 10570 926 902 902 q4 4657684 706 684 q5 7598288328862883 q6 190 123 125 123 q7 1189845 816 816 q8 9550212520332033 q9 7348652565086508 q10 8167252725022502 q11 422 238 216 216 q12 788 307 284 284 q13 17962 303929772977 q14 286 266 252 252 q15 504 455 450 450 q16 490 412 389 389 q17 943 559 574 559 q18 6731591358255825 q19 1592145114851451 q20 554 288 280 280 q21 7469319832723198 q22 811 306 306 306 Total cold run time: 107471 ms Total hot run time: 36920 ms - Round 2, with runtime_filter_mode=off - q1 4137407140984071 q2 325 223 222 222 q3 2947288628492849 q4 1833155815611558 q5 5249528852385238 q6 197 120 117 117 q7 2247183418501834 q8 3156330232993299 q9 8614859886058598 q10 6205363336423633 q11 536 441 441 441 q12 737 554 549 549 q13 10230 290028482848 q14 276 250 255 250 q15 479 449 449 449 q16 465 409 429 409 q17 1762149514541454 q18 7734722170667066 q19 5615152515371525 q20 1955171217151712 q21 4873474246624662 q22 520 458 467 458 Total cold run time: 70092 ms Total hot run time: 53242 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](binlog)Support inverted index in CCR (#31743) [doris]
doris-robot commented on PR #32101: URL: https://github.com/apache/doris/pull/32101#issuecomment-1990927962 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ui) fix database cannot be chosen bug [doris]
morningman commented on PR #32091: URL: https://github.com/apache/doris/pull/32091#issuecomment-1990928669 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](routine-load)add rack tag into fe & kafka routine load(#31531) [doris]
flashmouse commented on code in PR #32098: URL: https://github.com/apache/doris/pull/32098#discussion_r1520961237 ## fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java: ## @@ -93,6 +95,7 @@ public class KafkaRoutineLoadJob extends RoutineLoadJob { // kafka properties ,property prefix will be mapped to kafka custom parameters, which can be extended in the future private Map customProperties = Maps.newHashMap(); private Map convertedCustomProperties = Maps.newHashMap(); +private Map> convertedRackCustomProperties = Maps.newHashMap(); Review Comment: a new parameter **ConsumerConfig.CLIENT_RACK_CONFIG**(aka client.rack, this is a standard KafkaClient consumer config to specify the rack of the KafkaConsumer) introduced into convertedCustomProperties. so the value of each BE may not the same. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [improvement](binlog)Support inverted index in CCR (#31743) [doris]
qidaye opened a new pull request, #32101: URL: https://github.com/apache/doris/pull/32101 ## Proposed changes pick from #31743 ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](binlog)Support inverted index in CCR (#31743) [doris]
qidaye commented on PR #32101: URL: https://github.com/apache/doris/pull/32101#issuecomment-1990928166 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ui) fix database cannot be chosen bug [doris]
github-actions[bot] commented on PR #32091: URL: https://github.com/apache/doris/pull/32091#issuecomment-1990929386 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](ui) fix database cannot be chosen bug [doris]
github-actions[bot] commented on PR #32091: URL: https://github.com/apache/doris/pull/32091#issuecomment-1990929430 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Pick fix mow index compaction and revert comparePostings [doris-thirdparty]
qidaye merged PR #199: URL: https://github.com/apache/doris-thirdparty/pull/199 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](merge-cloud) Keep RecyclePartitionInfo compatible with the cloud mode [doris]
w41ter opened a new pull request, #32102: URL: https://github.com/apache/doris/pull/32102 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-thirdparty) branch clucene updated: Pick fix mow index compaction and revert comparePostings (#199)
This is an automated email from the ASF dual-hosted git repository. jianliangqi pushed a commit to branch clucene in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git The following commit(s) were added to refs/heads/clucene by this push: new ef95e67ae3 Pick fix mow index compaction and revert comparePostings (#199) ef95e67ae3 is described below commit ef95e67ae3123409f006072194f742a079603159 Author: qiye AuthorDate: Tue Mar 12 15:18:39 2024 +0800 Pick fix mow index compaction and revert comparePostings (#199) * [fix](write)revert comparePostings due to write core (#195) revert https://github.com/apache/doris-thirdparty/pull/156 * [fix](index compaction)Remove INT32_MAX out of destPostingQueues (#198) --- src/core/CLucene/index/IndexWriter.cpp | 101 + src/core/CLucene/index/SDocumentWriter.cpp | 26 +--- 2 files changed, 77 insertions(+), 50 deletions(-) diff --git a/src/core/CLucene/index/IndexWriter.cpp b/src/core/CLucene/index/IndexWriter.cpp index e30abf5107..0d770182ba 100644 --- a/src/core/CLucene/index/IndexWriter.cpp +++ b/src/core/CLucene/index/IndexWriter.cpp @@ -41,6 +41,7 @@ #include #include #include +#include #define FINALLY_CLOSE_OUTPUT(x) \ try { \ @@ -1667,34 +1668,47 @@ void IndexWriter::mergeTerms(bool hasProx) { std::vector> freqBuffers(numDestIndexes); auto destPostingQueues = _CLNEW postingQueue(matchSize); std::vector destDocs(matchSize); -for (int i = 0; i < matchSize; ++i) { -smi = match[i]; -TermPositions *postings = smi->getPositions(); -postings->seek(smi->termEnum); -if (postings->next()) { +auto processPostings = [&](TermPositions* postings, DestDoc* destDoc, int srcIdx) { +while (postings->next()) { int srcDoc = postings->doc(); std::pair p = _trans_vec[smi->readerIndex][srcDoc]; -uint32_t destIdx = p.first; -uint32_t destDocId = p.second; - -destDocs[i].srcIdx = i; -destDocs[i].destIdx = destIdx; -destDocs[i].destDocId = destDocId; +destDoc->destIdx = p.first; +destDoc->destDocId = p.second; +destDoc->srcIdx = srcIdx; +// indicates current row not exist in Doris dest segment. +// So we ignore this doc here. +if (destDoc->destIdx == UINT32_MAX || destDoc->destDocId == UINT32_MAX) { +if (infoStream != nullptr) { +std::stringstream ss; +ss << "skip UINT32_MAX, srcIdx: " << smi->readerIndex << ", srcDoc: " << srcDoc + << ", destIdx: " << destDoc->destIdx << ", destDocId: " << destDoc->destDocId; +message(ss.str()); +} +continue; +} if (hasProx) { int32_t freq = postings->freq(); -destDocs[i].destFreq = freq; -destDocs[i].destPositions.resize(freq); +destDoc->destFreq = freq; +destDoc->destPositions.resize(freq); for (int32_t j = 0; j < freq; j++) { int32_t position = postings->nextPosition(); -destDocs[i].destPositions[j] = position; +destDoc->destPositions[j] = position; } } -destPostingQueues->put(&destDocs[i]); +destPostingQueues->put(destDoc); +break; } +}; + +for (int i = 0; i < matchSize; ++i) { +smi = match[i]; +auto* postings = smi->getPositions(); +postings->seek(smi->termEnum); +processPostings(postings, &destDocs[i], i); } auto encode = [](IndexOutput* out, std::vector& buffer, bool isDoc) { @@ -1719,12 +1733,27 @@ void IndexWriter::mergeTerms(bool hasProx) { auto destDocId = destDoc->destDocId; auto destFreq = destDoc->destFreq; auto& descPositions = destDoc->destPositions; -// indicates current row not exist in Doris dest segment. -// So we ignore this doc here. -if (destIdx == UINT32_MAX || destDocId == UINT32_MAX) { -continue; +if (infoStream != nullptr) { +for (int i = 0; i < _trans_vec.size(); ++i) { +// find pair < destIdx, destDocId > in _trans_vec[i] to get the index of the pair +auto it = std::find_if(_trans_vec[i].begin(), _trans_vec[i].end(), + [destIdx, destDocId](const std::pai
Re: [PR] [fix](merge-cloud) Keep RecyclePartitionInfo compatible with the cloud mode [doris]
doris-robot commented on PR #32102: URL: https://github.com/apache/doris/pull/32102#issuecomment-1990930114 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (enhance)(Cache) Control whether writing into file cache for query and load [doris]
gavinchou commented on code in PR #32099: URL: https://github.com/apache/doris/pull/32099#discussion_r1520962375 ## be/src/olap/rowset/beta_rowset_reader.cpp: ## @@ -232,6 +232,8 @@ Status BetaRowsetReader::get_segment_iterators(RowsetReaderContext* read_context _read_options.io_ctx.query_id = &_read_context->runtime_state->query_id(); _read_options.io_ctx.read_file_cache = _read_context->runtime_state->query_options().enable_file_cache; +_read_options.io_ctx.is_disposable = + _read_context->runtime_state->query_options().disable_file_cache; Review Comment: reuse `_read_context->runtime_state->query_options().enable_file_cache` because `disable_file_cache` is deprecated. BTW remove `disable_file_cache` by renaming it to `DEPRECATED_disable_file_cache` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](merge-cloud) Keep RecyclePartitionInfo compatible with the cloud mode [doris]
w41ter commented on PR #32102: URL: https://github.com/apache/doris/pull/32102#issuecomment-1990930178 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (enhance)(Cache) Control whether writing into file cache for query and load [doris]
gavinchou commented on code in PR #32099: URL: https://github.com/apache/doris/pull/32099#discussion_r1520965317 ## be/src/olap/rowset/beta_rowset_writer.cpp: ## @@ -207,8 +207,11 @@ Status BetaRowsetWriter::_load_noncompacted_segment(segment_v2::SegmentSharedPtr } auto path = BetaRowset::segment_file_path(_context.rowset_dir, _context.rowset_id, segment_id); io::FileReaderOptions reader_options { -.cache_type = config::enable_file_cache ? io::FileCachePolicy::FILE_BLOCK_CACHE -: io::FileCachePolicy::NO_CACHE, +.cache_type = Review Comment: is enable_file_cache sufficient for the semantic? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](routine-load)add rack tag into fe & kafka routine load(#31531) [doris]
flashmouse commented on PR #32098: URL: https://github.com/apache/doris/pull/32098#issuecomment-1990937862 > Is rack-aware work when using routine load after this pr? after this pr, [KIP-392: Allow consumers to fetch from closest replica](https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica) would be effective, but we still could do some follow-up work to optimize FE assign partitions logic to get a better effect(ex [KIP-881: Rack-aware Partition Assignment for Kafka Consumers](https://cwiki.apache.org/confluence/display/KAFKA/KIP-881%3A+Rack-aware+Partition+Assignment+for+Kafka+Consumers)) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](binlog)Support inverted index in CCR (#31743) [doris]
github-actions[bot] commented on PR #32101: URL: https://github.com/apache/doris/pull/32101#issuecomment-1990937890 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](routine-load)add rack tag into fe & kafka routine load(#31531) [doris]
doris-robot commented on PR #32098: URL: https://github.com/apache/doris/pull/32098#issuecomment-1990938496 TPC-H: Total hot run time: 36906 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit afef16d0d72644a6eba18bd72ea571b8848721e4, data reload: false -- Round 1 -- q1 17843 419241004100 q2 2024157 146 146 q3 11204 920 953 920 q4 5267758 703 703 q5 7877279828292798 q6 187 129 126 126 q7 1252854 836 836 q8 9985213220872087 q9 7697644964556449 q10 8160250825542508 q11 428 214 230 214 q12 801 295 277 277 q13 17973 308429912991 q14 282 255 253 253 q15 495 454 454 454 q16 468 397 407 397 q17 944 533 632 533 q18 6654590559545905 q19 1555145113621362 q20 551 287 282 282 q21 7374327332613261 q22 796 308 304 304 Total cold run time: 109817 ms Total hot run time: 36906 ms - Round 2, with runtime_filter_mode=off - q1 4105405941624059 q2 321 226 222 222 q3 3002287128032803 q4 1853157315671567 q5 5189527452235223 q6 193 115 115 115 q7 2291185618431843 q8 3163328833043288 q9 8618855385788553 q10 6136369036693669 q11 534 440 441 440 q12 725 571 571 571 q13 11983 285228452845 q14 273 245 241 241 q15 481 456 438 438 q16 448 398 424 398 q17 1720149514831483 q18 7603728670777077 q19 1600151215471512 q20 1885169817481698 q21 4788477347374737 q22 515 427 437 427 Total cold run time: 67426 ms Total hot run time: 53209 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](partition) Fix partition id eq 0 impact publish [doris]
deardeng commented on PR #32072: URL: https://github.com/apache/doris/pull/32072#issuecomment-1990938875 这个pr先关了,打桩弄出3副本,2副本partition id = 0的,测试insert。 没用 这个pr修复,insert 之后是commit 状态,然后过5分钟后,变成visble 用这个pr修复的,insert之后是publish状态。也就是说pr修复本问题,是可以的。 但是感觉是头痛医头。本质上还是partition_id 和tabletinfo 的映射关系出了问题。 以后别人若用这个关系,可能引入别的问题。 pid=0的需要一个治本的方法。 所以这个pr先关了 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](Cloud) Add property storage vault for OlapTable [doris]
doris-robot commented on PR #31628: URL: https://github.com/apache/doris/pull/31628#issuecomment-1990939037 TPC-DS: Total hot run time: 186991 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 43c0b6563d19d69d4fb769f3a5bc168dbd33ef6c, data reload: false query1 945 369 341 341 query2 7411212119131913 query3 6708217 212 212 query4 31029 20625 20621 20621 query5 4351443 394 394 query6 266 176 171 171 query7 4625293 290 290 query8 226 166 165 165 query9 9959226222472247 query10 450 236 244 236 query11 16941 14166 14206 14166 query12 132 90 82 82 query13 1636413 415 413 query14 10844 10941 10442 10442 query15 252 209 200 200 query16 8020279 267 267 query17 1913572 568 568 query18 2034292 280 280 query19 212 169 155 155 query20 92 90 83 83 query21 204 129 124 124 query22 4661445343834383 query23 31879 31160 31145 31145 query24 10753 284927972797 query25 627 376 367 367 query26 1193157 168 157 query27 2403369 353 353 query28 7240188518871885 query29 880 630 649 630 query30 305 154 151 151 query31 990 722 734 722 query32 91 59 59 59 query33 775 259 263 259 query34 1093476 479 476 query35 828 606 619 606 query36 998 904 870 870 query37 113 75 75 75 query38 3280325231363136 query39 1423136614111366 query40 216 115 112 112 query41 49 54 48 48 query42 100 96 99 96 query43 505 449 451 449 query44 1258733 698 698 query45 279 270 256 256 query46 1094675 679 675 query47 1677162315731573 query48 455 364 353 353 query49 1124344 333 333 query50 739 371 371 371 query51 6696664565326532 query52 111 98 90 90 query53 341 276 283 276 query54 315 241 252 241 query55 82 77 79 77 query56 249 256 231 231 query57 1082100910341009 query58 230 207 203 203 query59 2820264926392639 query60 260 268 257 257 query61 115 113 110 110 query62 594 401 391 391 query63 304 278 283 278 query64 5683380737673767 query65 3070302629932993 query66 852 330 312 312 query67 15048 14618 14709 14618 query68 8575511 521 511 query69 630 383 376 376 query70 1289113711241124 query71 521 279 272 272 query72 6551265625472547 query73 749 313 317 313 query74 7813656265606560 query75 9388795281907952 query76 5154902 973 902 query77 675 258 252 252 query78 10323 966097259660 query79 13009 532 508 508 query80 2447383 358 358 query81 519 219 210 210 query82 526 204 217 204 query83 226 140 136 136 query84 284 80 78 78 query85 1295328 310 310 query86 374 298 280 280 query87 3421325732203220 query88 5175224622612246 query89 504 365 364 364 query90 2013176 171 171 query91 166 125 142 125 query92 59 46 47 46 query93 7286501 480 480 query94 1310177 173 173 query95 427 333 326 326 query96 609 265 261 261 query97 4406427942704270 query98 229 214 213 213 query99 1076759 748 748 Total cold run time: 320159 ms Total hot run time: 186991 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t
Re: [PR] [improve] Refactor block file cache and add more ut case [doris]
Lchangliang commented on code in PR #30039: URL: https://github.com/apache/doris/pull/30039#discussion_r1520969517 ## be/src/common/config.cpp: ## @@ -984,20 +984,12 @@ DEFINE_Bool(enable_file_cache, "false"); // format: [{"path":"/path/to/file_cache","total_size":21474836480,"query_limit":10737418240}] // format: [{"path":"/path/to/file_cache","total_size":21474836480,"query_limit":10737418240},{"path":"/path/to/file_cache2","total_size":21474836480,"query_limit":10737418240}] DEFINE_String(file_cache_path, ""); -DEFINE_Int64(file_cache_max_file_segment_size, "4194304"); // 4MB -// 4KB <= file_cache_max_file_segment_size <= 256MB -DEFINE_Validator(file_cache_max_file_segment_size, [](const int64_t config) -> bool { -return config >= 4096 && config <= 268435456; -}); -DEFINE_Int64(file_cache_min_file_segment_size, "1048576"); // 1MB -// 4KB <= file_cache_min_file_segment_size <= 256MB -DEFINE_Validator(file_cache_min_file_segment_size, [](const int64_t config) -> bool { -return config >= 4096 && config <= 268435456 && - config <= config::file_cache_max_file_segment_size; -}); +DEFINE_Int64(file_cache_each_block_size, "1048576"); // 1MB Review Comment: Have a check in be/src/runtime/exec_env_init.cpp:init_file_cache_factory -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](partition) Fix partition id eq 0 impact publish [doris]
deardeng closed pull request #32072: [Fix](partition) Fix partition id eq 0 impact publish URL: https://github.com/apache/doris/pull/32072 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](function) incorrect result of eq_for_null [doris]
mrhhsg opened a new pull request, #32103: URL: https://github.com/apache/doris/pull/32103 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](function) incorrect result of eq_for_null [doris]
doris-robot commented on PR #32103: URL: https://github.com/apache/doris/pull/32103#issuecomment-1990940642 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](function) incorrect result of eq_for_null [doris]
mrhhsg commented on PR #32103: URL: https://github.com/apache/doris/pull/32103#issuecomment-1990940852 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids): use table map to construct struct info [doris]
github-actions[bot] commented on PR #32058: URL: https://github.com/apache/doris/pull/32058#issuecomment-1990942880 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](Cloud) Add property storage vault for OlapTable [doris]
ByteYue commented on PR #31628: URL: https://github.com/apache/doris/pull/31628#issuecomment-1990945588 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [release](build script) fix build release bug when build multiple version on same machine [doris]
shuke987 opened a new pull request, #32104: URL: https://github.com/apache/doris/pull/32104 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [release](build script) fix build release bug when build multiple version on same machine [doris]
shuke987 commented on PR #32104: URL: https://github.com/apache/doris/pull/32104#issuecomment-1990945991 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](function) incorrect result of eq_for_null [doris]
github-actions[bot] commented on PR #32103: URL: https://github.com/apache/doris/pull/32103#issuecomment-1990946176 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [release](build script) fix build release bug when build multiple version on same machine [doris]
doris-robot commented on PR #32104: URL: https://github.com/apache/doris/pull/32104#issuecomment-1990945927 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](function) incorrect result of eq_for_null [doris]
github-actions[bot] commented on PR #32103: URL: https://github.com/apache/doris/pull/32103#issuecomment-1990946225 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](fe) Optimize calculate load job num metric in FE [doris]
doris-robot commented on PR #31952: URL: https://github.com/apache/doris/pull/31952#issuecomment-1990946232 TPC-H: Total hot run time: 36260 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b0bd19f3abaf65240db248096cfc2cd1b8dafc80, data reload: false -- Round 1 -- q1 17695 408540674067 q2 2012146 145 145 q3 10579 950 884 884 q4 4652697 671 671 q5 7569275327442744 q6 185 123 125 123 q7 1147822 802 802 q8 9479202819651965 q9 7173644264066406 q10 8133246024732460 q11 418 225 211 211 q12 788 303 294 294 q13 17963 297229662966 q14 268 249 249 249 q15 502 454 465 454 q16 472 390 399 390 q17 932 538 523 523 q18 6509579657835783 q19 1584141714621417 q20 527 272 281 272 q21 7365314432143144 q22 807 303 290 290 Total cold run time: 106759 ms Total hot run time: 36260 ms - Round 2, with runtime_filter_mode=off - q1 4093406540904065 q2 322 226 228 226 q3 2982277527642764 q4 1831151114851485 q5 5201524751925192 q6 193 121 117 117 q7 2207184218191819 q8 3140324132673241 q9 8474850885418508 q10 6066367936453645 q11 529 454 451 451 q12 726 541 528 528 q13 11243 286028582858 q14 273 266 258 258 q15 487 452 446 446 q16 455 416 407 407 q17 1720147214611461 q18 7632720270827082 q19 6534149514971495 q20 1913169617181696 q21 4873469847844698 q22 559 463 467 463 Total cold run time: 71453 ms Total hot run time: 52905 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](merge-cloud) Keep PartitionInfo compatible with the cloud mode [doris]
github-actions[bot] commented on PR #32008: URL: https://github.com/apache/doris/pull/32008#issuecomment-1990947160 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](merge-cloud) Keep RecyclePartitionInfo compatible with the cloud mode [doris]
github-actions[bot] commented on PR #32102: URL: https://github.com/apache/doris/pull/32102#issuecomment-1990947323 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [bugfix](becore) be will core when stop because the map is modified during iterator [doris]
yiguolei opened a new pull request, #32105: URL: https://github.com/apache/doris/pull/32105 ## Proposed changes = ==9396==ERROR: AddressSanitizer: heap-use-after-free on address 0x6030052f26e0 at pc 0x563a94549f30 bp 0x7ffd8ec8eb90 sp 0x7ffd8ec8eb88 READ of size 8 at 0x6030052f26e0 thread T0 #0 0x563a94549f2f in std::__shared_ptr::get() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291:16 #1 0x563a94549ee6 in std::__shared_ptr_access::_M_get() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:990:66 #2 0x563a94549166 in std::__shared_ptr_access::operator->() const /var/local/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:984:9 #3 0x563aa975e2f9 in doris::vectorized::VDataStreamMgr::~VDataStreamMgr() /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_mgr.cpp:46:9 #4 0x563a7b426e4f in doris::ExecEnv::destroy() /home/zcp/repo_center/doris_master/doris/be/src/runtime/exec_env_init.cpp:630:5 #5 0x563a77e67597 in main /home/zcp/repo_center/doris_master/doris/be/src/service/doris_main.cpp:597:15 #6 0x7f7a38f45082 in __libc_start_main /build/glibc-SzIz7B/glibc-2.31/csu/../csu/libc-start.c:308:16 #7 0x563a77d8a029 in _start (/mnt/hdd01/ci/master-deploy/be/lib/doris_be+0x25e4f029) (BuildId: ad37d7448f997131) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](becore) be will core when stop because the map is modified during iterator [doris]
doris-robot commented on PR #32105: URL: https://github.com/apache/doris/pull/32105#issuecomment-1990947949 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] (enhance)(Cache) Control whether writing into file cache for query and load [doris]
ByteYue commented on code in PR #32099: URL: https://github.com/apache/doris/pull/32099#discussion_r1520974438 ## be/src/olap/rowset/beta_rowset_reader.cpp: ## @@ -232,6 +232,8 @@ Status BetaRowsetReader::get_segment_iterators(RowsetReaderContext* read_context _read_options.io_ctx.query_id = &_read_context->runtime_state->query_id(); _read_options.io_ctx.read_file_cache = _read_context->runtime_state->query_options().enable_file_cache; +_read_options.io_ctx.is_disposable = + _read_context->runtime_state->query_options().disable_file_cache; Review Comment: 目前disable_file_cache这个名字在社区还不能一下子废除,在导入路径是否写cache的时候会用到 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](partition) Fix partition id eq 0 impact publish [doris]
deardeng commented on PR #32072: URL: https://github.com/apache/doris/pull/32072#issuecomment-1990948847 用此pr修了,是这样的  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](partition) Fix partition id eq 0 impact publish [doris]
deardeng commented on PR #32072: URL: https://github.com/apache/doris/pull/32072#issuecomment-1990948091  没用那个pr修的时候,3副本2副本(pid=0),insert是这样的 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](becore) be will core when stop because the map is modified during iterator [doris]
yiguolei commented on PR #32105: URL: https://github.com/apache/doris/pull/32105#issuecomment-1990948530 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](Cloud) Add property storage vault for OlapTable [doris]
github-actions[bot] commented on PR #31628: URL: https://github.com/apache/doris/pull/31628#issuecomment-1990950663 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](Cloud) Add property storage vault for OlapTable [doris]
github-actions[bot] commented on PR #31628: URL: https://github.com/apache/doris/pull/31628#issuecomment-1990950700 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](binlog)Support inverted index in CCR (#31743) [doris]
yiguolei merged PR #32101: URL: https://github.com/apache/doris/pull/32101 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [improvement](binlog)Support inverted index in CCR (#31743) (#32101)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/branch-2.1 by this push: new 5f125bbaaa1 [improvement](binlog)Support inverted index in CCR (#31743) (#32101) 5f125bbaaa1 is described below commit 5f125bbaaa1ddc0b8b98a2043c9c05b70a8c2472 Author: qiye AuthorDate: Tue Mar 12 15:34:08 2024 +0800 [improvement](binlog)Support inverted index in CCR (#31743) (#32101) --- be/src/http/action/download_binlog_action.cpp | 39 be/src/olap/rowset/beta_rowset.cpp| 44 - be/src/olap/snapshot_manager.cpp | 53 ++- be/src/olap/tablet.cpp| 26 - be/src/olap/tablet.h | 5 + be/src/olap/tablet_manager.cpp| 17 +++- be/src/olap/task/engine_clone_task.cpp| 14 ++- be/src/service/backend_service.cpp| 132 -- 8 files changed, 308 insertions(+), 22 deletions(-) diff --git a/be/src/http/action/download_binlog_action.cpp b/be/src/http/action/download_binlog_action.cpp index 697512b2a30..dbe2880d3b4 100644 --- a/be/src/http/action/download_binlog_action.cpp +++ b/be/src/http/action/download_binlog_action.cpp @@ -47,6 +47,7 @@ const std::string kTabletIdParameter = "tablet_id"; const std::string kBinlogVersionParameter = "binlog_version"; const std::string kRowsetIdParameter = "rowset_id"; const std::string kSegmentIndexParameter = "segment_index"; +const std::string kSegmentIndexIdParameter = "segment_index_id"; // get http param, if no value throw exception const auto& get_http_param(HttpRequest* req, const std::string& param_name) { @@ -130,6 +131,42 @@ void handle_get_segment_file(HttpRequest* req, bufferevent_rate_limit_group* rat do_file_response(segment_file_path, req, rate_limit_group); } +/// handle get segment index file, need tablet_id, rowset_id, segment_index && segment_index_id +void handle_get_segment_index_file(HttpRequest* req, + bufferevent_rate_limit_group* rate_limit_group) { +// Step 1: get download file path +std::string segment_index_file_path; +try { +const auto& tablet_id = get_http_param(req, kTabletIdParameter); +auto tablet = get_tablet(tablet_id); +const auto& rowset_id = get_http_param(req, kRowsetIdParameter); +const auto& segment_index = get_http_param(req, kSegmentIndexParameter); +const auto& segment_index_id = req->param(kSegmentIndexIdParameter); +segment_index_file_path = +tablet->get_segment_index_filepath(rowset_id, segment_index, segment_index_id); +} catch (const std::exception& e) { +HttpChannel::send_reply(req, HttpStatus::INTERNAL_SERVER_ERROR, e.what()); +LOG(WARNING) << "get download file path failed, error: " << e.what(); +return; +} + +// Step 2: handle download +// check file exists +bool exists = false; +Status status = io::global_local_filesystem()->exists(segment_index_file_path, &exists); +if (!status.ok()) { +HttpChannel::send_reply(req, HttpStatus::INTERNAL_SERVER_ERROR, status.to_string()); +LOG(WARNING) << "check file exists failed, error: " << status.to_string(); +return; +} +if (!exists) { +HttpChannel::send_reply(req, HttpStatus::NOT_FOUND, "file not exist."); +LOG(WARNING) << "file not exist, file path: " << segment_index_file_path; +return; +} +do_file_response(segment_index_file_path, req, rate_limit_group); +} + void handle_get_rowset_meta(HttpRequest* req) { try { const auto& tablet_id = get_http_param(req, kTabletIdParameter); @@ -183,6 +220,8 @@ void DownloadBinlogAction::handle(HttpRequest* req) { handle_get_binlog_info(req); } else if (method == "get_segment_file") { handle_get_segment_file(req, _rate_limit_group.get()); +} else if (method == "get_segment_index_file") { +handle_get_segment_index_file(req, _rate_limit_group.get()); } else if (method == "get_rowset_meta") { handle_get_rowset_meta(req); } else { diff --git a/be/src/olap/rowset/beta_rowset.cpp b/be/src/olap/rowset/beta_rowset.cpp index e3f28726c4d..58e495be222 100644 --- a/be/src/olap/rowset/beta_rowset.cpp +++ b/be/src/olap/rowset/beta_rowset.cpp @@ -439,7 +439,7 @@ Status BetaRowset::add_to_binlog() { if (fs->type() != io::FileSystemType::LOCAL) { return Status::InternalError("should be local file system"); } -io::LocalFileSystem* local_fs = static_cast(fs.get()); +auto* local_fs = static_cast(fs.get()); // all segments are in the same directory, so cache binlog_dir without multi times check std::string binlog_dir; @@ -447,6 +447,22 @@ Status BetaRowset::add_to_binlog() { auto segments
Re: [PR] [feature](routine-load)add rack tag into fe & kafka routine load(#31531) [doris]
doris-robot commented on PR #32098: URL: https://github.com/apache/doris/pull/32098#issuecomment-199093 TPC-DS: Total hot run time: 187436 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit afef16d0d72644a6eba18bd72ea571b8848721e4, data reload: false query1 912 361 352 352 query2 7798200519661966 query3 6701209 207 207 query4 31120 20711 20707 20707 query5 4421474 405 405 query6 255 179 169 169 query7 4627297 288 288 query8 229 173 183 173 query9 9973235723202320 query10 455 234 242 234 query11 17313 14129 14167 14129 query12 130 85 108 85 query13 1622419 421 419 query14 11247 11239 10713 10713 query15 279 218 195 195 query16 8021264 262 262 query17 1939559 547 547 query18 2079298 287 287 query19 288 154 151 151 query20 94 85 85 85 query21 204 132 129 129 query22 4773450944684468 query23 31852 30847 31339 30847 query24 10688 284028292829 query25 638 372 390 372 query26 1410161 155 155 query27 2999353 354 353 query28 8023187419031874 query29 888 663 632 632 query30 303 153 150 150 query31 967 700 736 700 query32 95 57 60 57 query33 774 263 246 246 query34 1082482 484 482 query35 829 626 619 619 query36 997 879 854 854 query37 124 76 76 76 query38 3378318732123187 query39 1438136513641364 query40 216 110 109 109 query41 51 47 48 47 query42 105 94 97 94 query43 481 437 445 437 query44 1267740 712 712 query45 280 255 260 255 query46 1096679 712 679 query47 1700161915981598 query48 441 355 364 355 query49 1131341 374 341 query50 766 373 375 373 query51 6574671965076507 query52 116 95 90 90 query53 370 280 276 276 query54 309 232 245 232 query55 81 81 80 80 query56 256 226 236 226 query57 1105103410191019 query58 230 204 211 204 query59 2749264926602649 query60 266 292 280 280 query61 122 109 111 109 query62 612 386 394 386 query63 296 268 272 268 query64 6570370337913703 query65 3089301230173012 query66 1395314 321 314 query67 15098 14936 14588 14588 query68 8649531 548 531 query69 613 378 370 370 query70 1269119710721072 query71 529 289 276 276 query72 6634261824802480 query73 730 325 316 316 query74 8050660066016600 query75 10018 807379707970 query76 5021919 970 919 query77 644 301 265 265 query78 10456 976597259725 query79 9449522 523 522 query80 1570385 433 385 query81 511 206 206 206 query82 887 202 197 197 query83 212 143 146 143 query84 281 76 75 75 query85 1453329 310 310 query86 454 276 289 276 query87 3381326932293229 query88 4986234723422342 query89 511 367 382 367 query90 1978172 175 172 query91 155 127 150 127 query92 64 46 54 46 query93 6846525 501 501 query94 177 175 175 query95 424 337 332 332 query96 600 274 273 273 query97 4419425842804258 query98 235 210 203 203 query99 1257742 747 742 Total cold run time: 321134 ms Total hot run time: 187436 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t
Re: [PR] [feature](merge-cloud) implement alter table property of cloud mode [doris]
dataroaring merged PR #31887: URL: https://github.com/apache/doris/pull/31887 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (7f665f381ac -> 5985044be28)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 7f665f381ac [docs](docs) Update release note 2.0.6 and readme of Master branch (#32087) add 5985044be28 [feature](merge-cloud) implement alter table property of cloud mode (#31887) No new revisions were added by this update. Summary of changes: cloud/src/meta-service/meta_service.cpp| 4 + .../main/java/org/apache/doris/common/Config.java | 3 + .../main/java/org/apache/doris/alter/Alter.java| 30 +-- .../apache/doris/alter/SchemaChangeHandler.java| 18 ++ .../cloud/alter/CloudSchemaChangeHandler.java | 256 + gensrc/proto/cloud.proto | 2 + gensrc/proto/olap_file.proto | 6 +- 7 files changed, 289 insertions(+), 30 deletions(-) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/cloud/alter/CloudSchemaChangeHandler.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [release](build script) fix build release bug when build multiple version on same machine [doris]
github-actions[bot] commented on PR #32104: URL: https://github.com/apache/doris/pull/32104#issuecomment-1990959719 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [release](build script) fix build release bug when build multiple version on same machine [doris]
github-actions[bot] commented on PR #32104: URL: https://github.com/apache/doris/pull/32104#issuecomment-1990959750 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [release](build script) fix build release bug when build multiple version on same machine [doris]
yiguolei merged PR #32104: URL: https://github.com/apache/doris/pull/32104 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (5985044be28 -> 6c6466b465d)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 5985044be28 [feature](merge-cloud) implement alter table property of cloud mode (#31887) add 6c6466b465d [release](build script) fix build release bug when build multiple version on same machine #32104 No new revisions were added by this update. Summary of changes: build-for-release.sh | 1 + 1 file changed, 1 insertion(+) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](fe) Optimize calculate load job num metric in FE [doris]
doris-robot commented on PR #31952: URL: https://github.com/apache/doris/pull/31952#issuecomment-1990960965 TPC-DS: Total hot run time: 186493 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit b0bd19f3abaf65240db248096cfc2cd1b8dafc80, data reload: false query1 924 372 346 346 query2 7410199419241924 query3 6709213 208 208 query4 31365 20765 20664 20664 query5 4420412 390 390 query6 255 169 162 162 query7 4619302 284 284 query8 224 168 169 168 query9 9298229722832283 query10 452 235 256 235 query11 17141 14145 14189 14145 query12 139 93 81 81 query13 1628409 415 409 query14 11381 10584 10438 10438 query15 260 204 193 193 query16 7977271 257 257 query17 1917566 529 529 query18 2067282 279 279 query19 225 157 151 151 query20 89 84 86 84 query21 201 136 124 124 query22 4677446044114411 query23 31656 31182 31266 31182 query24 10790 288127842784 query25 624 371 382 371 query26 1167160 163 160 query27 2526357 352 352 query28 7602191518931893 query29 878 637 621 621 query30 303 148 149 148 query31 980 713 729 713 query32 91 62 54 54 query33 767 254 254 254 query34 1083459 462 459 query35 825 593 595 593 query36 1000864 863 863 query37 121 75 76 75 query38 3266315431513151 query39 1396138213561356 query40 208 112 110 110 query41 50 46 46 46 query42 101 94 97 94 query43 485 457 460 457 query44 1255724 688 688 query45 266 259 258 258 query46 1084680 679 679 query47 1688157416071574 query48 458 359 349 349 query49 1137341 326 326 query50 757 375 373 373 query51 6704655366216553 query52 111 89 96 89 query53 339 269 273 269 query54 306 235 239 235 query55 82 78 80 78 query56 242 248 230 230 query57 1097100010031000 query58 237 199 202 199 query59 2620261625112511 query60 269 267 245 245 query61 110 111 110 110 query62 594 405 388 388 query63 296 271 279 271 query64 5551378338473783 query65 3055301230143012 query66 871 322 320 320 query67 14723 14413 14322 14322 query68 8621520 533 520 query69 630 378 366 366 query70 1259108911171089 query71 518 271 279 271 query72 6598263524982498 query73 732 312 318 312 query74 7989653966186539 query75 9792808178737873 query76 5155893 1006893 query77 689 268 251 251 query78 10349 964696449644 query79 9696514 506 506 query80 1635393 369 369 query81 521 208 210 208 query82 493 200 197 197 query83 220 141 148 141 query84 284 76 79 76 query85 1181348 313 313 query86 361 286 294 286 query87 3417329632213221 query88 4787236823522352 query89 489 373 359 359 query90 2071174 174 174 query91 160 130 136 130 query92 67 47 49 47 query93 6722509 484 484 query94 1295185 184 184 query95 440 333 332 332 query96 611 276 280 276 query97 4432426942624262 query98 241 213 211 211 query99 1034746 750 746 Total cold run time: 315638 ms Total hot run time: 186493 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t
Re: [PR] [feat](Nereids): use table map to construct struct info [doris]
github-actions[bot] commented on PR #32058: URL: https://github.com/apache/doris/pull/32058#issuecomment-1990961289 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [enhancement](Nereids) refine and speedup analyzer [doris]
924060929 opened a new pull request, #32106: URL: https://github.com/apache/doris/pull/32106 ## Proposed changes cherry from #31792 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](Nereids) refine and speedup analyzer [doris]
doris-robot commented on PR #32106: URL: https://github.com/apache/doris/pull/32106#issuecomment-1990962008 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](routine-load)add rack tag into fe & kafka routine load(#31531) [doris]
doris-robot commented on PR #32098: URL: https://github.com/apache/doris/pull/32098#issuecomment-1990962153 ClickBench: Total hot run time: 30 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit afef16d0d72644a6eba18bd72ea571b8848721e4, data reload: false query1 0.040.040.03 query2 0.060.030.03 query3 0.230.060.06 query4 1.670.100.10 query5 0.500.520.51 query6 1.180.600.60 query7 0.020.010.01 query8 0.030.030.02 query9 0.560.500.50 query10 0.540.530.54 query11 0.110.090.10 query12 0.110.090.09 query13 0.560.570.57 query14 0.760.750.73 query15 0.800.770.78 query16 0.390.400.39 query17 1.031.020.99 query18 0.190.270.22 query19 1.841.831.79 query20 0.020.010.01 query21 15.45 0.540.55 query22 2.591.761.61 query23 17.08 0.730.68 query24 4.271.631.06 query25 0.280.180.03 query26 0.620.130.13 query27 0.050.030.03 query28 10.72 0.850.86 query29 12.66 3.173.03 query30 0.660.620.54 query31 2.800.320.33 query32 3.410.450.45 query33 2.792.862.77 query34 16.08 4.304.38 query35 4.334.344.30 query36 1.071.021.03 query37 0.080.050.05 query38 0.050.030.03 query39 0.020.020.02 query40 0.170.140.16 query41 0.080.020.02 query42 0.030.020.02 query43 0.020.020.02 Total cold run time: 105.95 s Total hot run time: 30 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](nereids)EliminateSemiJoin should consider empty table [doris]
starocean999 opened a new pull request, #32107: URL: https://github.com/apache/doris/pull/32107 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)EliminateSemiJoin should consider empty table [doris]
doris-robot commented on PR #32107: URL: https://github.com/apache/doris/pull/32107#issuecomment-1990965974 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](nereids)EliminateSemiJoin should consider empty table [doris]
starocean999 commented on PR #32107: URL: https://github.com/apache/doris/pull/32107#issuecomment-1990966108 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](routine-load)add rack tag into fe & kafka routine load(#31531) [doris]
doris-robot commented on PR #32098: URL: https://github.com/apache/doris/pull/32098#issuecomment-1990966222 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit afef16d0d72644a6eba18bd72ea571b8848721e4 with default session variables Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 17.3 seconds inserted 1000 Rows, about 578K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Feature] Doris support Arrow Flight SQL protocol [doris]
jpohanka commented on issue #25514: URL: https://github.com/apache/doris/issues/25514#issuecomment-1990966353 @xinyiZzz Thank you for creating this feature request and for implementing Arrow Flight in Doris. Currently (Doris 2.1.0) we have only data retrieval. Are you also planning to implement data ingestion via Arrow Flight? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](fe) Optimize calculate load job num metric in FE [doris]
doris-robot commented on PR #31952: URL: https://github.com/apache/doris/pull/31952#issuecomment-1990967928 ClickBench: Total hot run time: 29.94 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit b0bd19f3abaf65240db248096cfc2cd1b8dafc80, data reload: false query1 0.040.030.03 query2 0.060.030.03 query3 0.230.060.07 query4 1.650.090.09 query5 0.510.500.50 query6 1.220.610.63 query7 0.020.020.01 query8 0.040.030.03 query9 0.560.500.50 query10 0.540.560.53 query11 0.110.090.09 query12 0.110.090.09 query13 0.570.570.56 query14 0.740.750.75 query15 0.800.780.78 query16 0.400.400.40 query17 0.981.011.04 query18 0.190.250.22 query19 1.881.671.81 query20 0.020.010.01 query21 15.40 0.550.55 query22 2.524.242.02 query23 17.09 0.850.75 query24 3.291.920.46 query25 0.110.200.03 query26 0.710.130.14 query27 0.020.030.05 query28 11.22 0.880.84 query29 12.53 3.133.21 query30 0.640.610.60 query31 2.800.340.36 query32 3.350.420.44 query33 2.792.762.85 query34 16.09 4.324.29 query35 4.314.324.35 query36 1.081.021.01 query37 0.070.060.05 query38 0.040.030.03 query39 0.030.020.02 query40 0.170.130.14 query41 0.070.020.02 query42 0.030.020.02 query43 0.030.030.02 Total cold run time: 105.06 s Total hot run time: 29.94 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](merge-cloud) fix no cluster for common user [doris]
github-actions[bot] commented on PR #32097: URL: https://github.com/apache/doris/pull/32097#issuecomment-1990968275 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [enhancement](test) exclude somecases not support in cloud mode [doris]
dataroaring opened a new pull request, #32108: URL: https://github.com/apache/doris/pull/32108 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](test) exclude somecases not support in cloud mode [doris]
doris-robot commented on PR #32108: URL: https://github.com/apache/doris/pull/32108#issuecomment-1990968573 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](test) exclude somecases not support in cloud mode [doris]
dataroaring commented on PR #32108: URL: https://github.com/apache/doris/pull/32108#issuecomment-1990968660 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](merge-cloud) Keep RecyclePartitionInfo compatible with the cloud mode [doris]
doris-robot commented on PR #32102: URL: https://github.com/apache/doris/pull/32102#issuecomment-1990968845 TPC-H: Total hot run time: 36640 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit acfcd2f0fbc3d5f279a2a9ca26b2a8fdd5538b2c, data reload: false -- Round 1 -- q1 17648 425841334133 q2 2057161 153 153 q3 10635 924 912 912 q4 4657713 689 689 q5 7594274127512741 q6 184 126 126 126 q7 1222817 804 804 q8 9500207320532053 q9 7803650564936493 q10 8116248124582458 q11 427 217 214 214 q12 785 304 298 298 q13 17959 303530023002 q14 276 248 264 248 q15 495 464 451 451 q16 466 404 392 392 q17 946 549 537 537 q18 6749599257895789 q19 1582134114471341 q20 550 291 294 291 q21 7343327332133213 q22 799 304 302 302 Total cold run time: 107793 ms Total hot run time: 36640 ms - Round 2, with runtime_filter_mode=off - q1 4116408541074085 q2 326 226 224 224 q3 2979284628922846 q4 1817159715381538 q5 5236524352525243 q6 198 115 118 115 q7 2290187018201820 q8 3149329032693269 q9 8558853485328532 q10 6162367236393639 q11 538 441 452 441 q12 739 527 536 527 q13 13271 283928612839 q14 280 245 255 245 q15 481 442 449 442 q16 453 410 407 407 q17 1747149014431443 q18 7786730970037003 q19 5542153615501536 q20 1908170717151707 q21 4888467848164678 q22 522 461 463 461 Total cold run time: 72986 ms Total hot run time: 53040 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](Nereids) refine and speedup analyzer [doris]
github-actions[bot] commented on PR #32106: URL: https://github.com/apache/doris/pull/32106#issuecomment-1990971040 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](function) incorrect result of eq_for_null [doris]
doris-robot commented on PR #32103: URL: https://github.com/apache/doris/pull/32103#issuecomment-1990971463 TeamCity be ut coverage result: Function Coverage: 34.98% (8556/24459) Line Coverage: 26.82% (69528/259264) Region Coverage: 26.06% (36074/138420) Branch Coverage: 23.03% (18437/80072) Coverage Report: http://coverage.selectdb-in.cc/coverage/1c4fcdcd5f410df88e911f06a40363ede0bbf9fe_1c4fcdcd5f410df88e911f06a40363ede0bbf9fe/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](fe) Optimize calculate load job num metric in FE [doris]
doris-robot commented on PR #31952: URL: https://github.com/apache/doris/pull/31952#issuecomment-1990972154 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit b0bd19f3abaf65240db248096cfc2cd1b8dafc80 with default session variables Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s Stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s Insert into select: 17.1 seconds inserted 1000 Rows, about 584K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](Nereids) Optimize expression [doris]
doris-robot commented on PR #32067: URL: https://github.com/apache/doris/pull/32067#issuecomment-1990972748 TPC-H: Total hot run time: 36333 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 3d75bded1c5c70216bbf403565539c521a234df1, data reload: false -- Round 1 -- q1 17623 412141124112 q2 2032151 144 144 q3 10569 913 883 883 q4 4650647 683 647 q5 7608270627412706 q6 185 127 129 127 q7 1171814 797 797 q8 9470198320431983 q9 7370649564126412 q10 8271250324712471 q11 429 227 207 207 q12 806 300 279 279 q13 17959 301030043004 q14 277 275 249 249 q15 498 446 442 442 q16 501 385 395 385 q17 926 571 534 534 q18 6671582457325732 q19 1543147713921392 q20 535 269 275 269 q21 7342328832593259 q22 796 299 326 299 Total cold run time: 107232 ms Total hot run time: 36333 ms - Round 2, with runtime_filter_mode=off - q1 4107404941214049 q2 318 233 224 224 q3 2952283528412835 q4 1821146715251467 q5 5153519352255193 q6 192 115 120 115 q7 2224186118261826 q8 3135325132823251 q9 8523850084828482 q10 6161370336403640 q11 536 445 435 435 q12 736 548 562 548 q13 13564 286128412841 q14 272 248 251 248 q15 482 448 437 437 q16 447 389 397 389 q17 1714148014701470 q18 7629718870967096 q19 5904156215571557 q20 1904168816971688 q21 4750458946244589 q22 534 445 449 445 Total cold run time: 73058 ms Total hot run time: 52825 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature-wip](Cloud) Add property storage vault for OlapTable [doris]
doris-robot commented on PR #31628: URL: https://github.com/apache/doris/pull/31628#issuecomment-1990974753 TeamCity be ut coverage result: Function Coverage: 34.97% (8556/24465) Line Coverage: 26.81% (69531/259305) Region Coverage: 26.07% (36087/138436) Branch Coverage: 23.03% (18443/80084) Coverage Report: http://coverage.selectdb-in.cc/coverage/c851ce8263db2a5b7696ee173eaab768d8fd1f02_c851ce8263db2a5b7696ee173eaab768d8fd1f02/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](Nereids) refine and speedup analyzer [doris]
yiguolei merged PR #32106: URL: https://github.com/apache/doris/pull/32106 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](Nereids): use table map to construct struct info [doris]
jackwener merged PR #32058: URL: https://github.com/apache/doris/pull/32058 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (6c6466b465d -> 5242a80d2bf)
This is an automated email from the ASF dual-hosted git repository. jakevin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 6c6466b465d [release](build script) fix build release bug when build multiple version on same machine #32104 add 5242a80d2bf [feat](Nereids): use table map to construct struct info (#32058) No new revisions were added by this update. Summary of changes: .../apache/doris/nereids/memo/StructInfoMap.java | 46 - .../doris/nereids/memo/StructInfoMapTest.java | 48 ++ .../apache/doris/utframe/TestWithFeService.java| 25 +++ 3 files changed, 108 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](becore) be will core when stop because the map is modified during iterator [doris]
doris-robot commented on PR #32105: URL: https://github.com/apache/doris/pull/32105#issuecomment-1990975714 TeamCity be ut coverage result: Function Coverage: 34.98% (8556/24459) Line Coverage: 26.82% (69523/259268) Region Coverage: 26.06% (36072/138423) Branch Coverage: 23.03% (18440/80074) Coverage Report: http://coverage.selectdb-in.cc/coverage/c900b0973e9cc5ee662613864be5bc97e7297562_c900b0973e9cc5ee662613864be5bc97e7297562/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](routine-load)add rack tag into fe & kafka routine load(#31531) [doris]
HHoflittlefish777 commented on code in PR #32098: URL: https://github.com/apache/doris/pull/32098#discussion_r1521001467 ## fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaTaskInfo.java: ## @@ -86,7 +87,11 @@ public TRoutineLoadTask createRoutineLoadTask() throws UserException { tKafkaLoadInfo.setTopic(routineLoadJob.getTopic()); tKafkaLoadInfo.setBrokers(routineLoadJob.getBrokerList()); tKafkaLoadInfo.setPartitionBeginOffset(partitionIdToOffset); - tKafkaLoadInfo.setProperties(routineLoadJob.getConvertedCustomProperties()); +Backend backend = Env.getCurrentSystemInfo().getBackend(beId); +if (backend == null) { +throw new UserException("failed to get be:" + beId + " rack because not exist"); Review Comment: If backend is null, can we choose other one? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [build]( build for release ) fix build bug when build multiple release version on same machine [doris]
shuke987 opened a new pull request, #32109: URL: https://github.com/apache/doris/pull/32109 …sion on same machine #32104 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [build]( build for release ) fix build bug when build multiple release version on same machine [doris]
doris-robot commented on PR #32109: URL: https://github.com/apache/doris/pull/32109#issuecomment-1990980764 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [build]( build for release ) fix build bug when build multiple release version on same machine [doris]
shuke987 commented on PR #32109: URL: https://github.com/apache/doris/pull/32109#issuecomment-1990981165 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](routine-load)add rack tag into fe & kafka routine load(#31531) [doris]
HHoflittlefish777 commented on code in PR #32098: URL: https://github.com/apache/doris/pull/32098#discussion_r1521002433 ## fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java: ## @@ -741,7 +750,8 @@ public boolean hasMoreDataToConsume(UUID taskId, Map partitionIdT // all offsets to be consumed are newer than offsets in cachedPartitionWithLatestOffsets, // maybe the cached offset is out-of-date, fetch from kafka server again List> tmp = KafkaUtil.getLatestOffsets(id, taskId, getBrokerList(), -getTopic(), getConvertedCustomProperties(), Lists.newArrayList(partitionIdToOffset.keySet())); +getTopic(), getConvertedCustomProperties(Tag.VALUE_DEFAULT_TAG), Review Comment: Is always Tag.VALUE_DEFAULT_TAG? ## fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java: ## @@ -741,7 +750,8 @@ public boolean hasMoreDataToConsume(UUID taskId, Map partitionIdT // all offsets to be consumed are newer than offsets in cachedPartitionWithLatestOffsets, // maybe the cached offset is out-of-date, fetch from kafka server again List> tmp = KafkaUtil.getLatestOffsets(id, taskId, getBrokerList(), -getTopic(), getConvertedCustomProperties(), Lists.newArrayList(partitionIdToOffset.keySet())); +getTopic(), getConvertedCustomProperties(Tag.VALUE_DEFAULT_TAG), Review Comment: Is it always Tag.VALUE_DEFAULT_TAG? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](merge-cloud) fix no cluster for common user [doris]
liaoxin01 commented on PR #32097: URL: https://github.com/apache/doris/pull/32097#issuecomment-1990982753 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (86bddefe6ba -> 5f125bbaaa1)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git discard 86bddefe6ba [enhancement](Nereids) refine and speedup analyzer (#31792) (#32106) This update removed existing revisions from the reference, leaving the reference pointing at a previous point in the repository history. * -- * -- N refs/heads/branch-2.1 (5f125bbaaa1) \ O -- O -- O (86bddefe6ba) Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: be/src/clucene |2 +- docs/en/docs/admin-manual/config/fe-config.md | 5471 +--- docs/en/docs/advanced/variables.md | 2945 ++- docs/zh-CN/docs/admin-manual/config/fe-config.md | 5457 +-- docs/zh-CN/docs/advanced/variables.md | 2766 ++ .../org/apache/doris/catalog/FunctionRegistry.java | 22 +- .../apache/doris/catalog/FunctionSignature.java|3 +- .../org/apache/doris/nereids/CascadesContext.java |9 - .../doris/nereids/analyzer/ComplexDataType.java| 22 - .../apache/doris/nereids/analyzer/MappingSlot.java | 115 - .../org/apache/doris/nereids/analyzer/Scope.java | 37 +- .../nereids/jobs/rewrite/PlanTreeRewriteJob.java | 18 +- .../rules/analysis/AvgDistinctToSumDivCount.java |2 +- .../nereids/rules/analysis/BindExpression.java | 1223 ++--- .../nereids/rules/analysis/BindSlotWithPaths.java |2 +- .../nereids/rules/analysis/CheckAnalysis.java | 12 +- .../rules/analysis/EliminateLogicalSelectHint.java |2 +- .../nereids/rules/analysis/ExpressionAnalyzer.java | 807 --- .../doris/nereids/rules/analysis/SlotBinder.java |2 +- .../nereids/rules/analysis/SubExprAnalyzer.java| 19 +- .../rewrite/PushDownLimitDistinctThroughUnion.java |2 +- .../rules/rewrite/PushProjectIntoUnion.java|2 +- .../rules/rewrite/PushProjectThroughUnion.java |2 +- .../mv/SelectMaterializedIndexWithAggregate.java |2 +- .../doris/nereids/trees/AbstractTreeNode.java | 10 +- .../org/apache/doris/nereids/trees/TreeNode.java | 17 +- .../doris/nereids/trees/expressions/CaseWhen.java | 31 +- .../nereids/trees/expressions/Expression.java | 65 +- .../nereids/trees/expressions/SlotReference.java |4 +- .../functions/ComputeSignatureHelper.java | 35 +- .../functions/ExplicitlyCastableSignature.java |4 - .../expressions/functions/IdenticalSignature.java |4 - .../functions/ImplicitlyCastableSignature.java | 10 +- .../functions/NullOrIdenticalSignature.java|4 - .../visitor/DefaultExpressionRewriter.java | 48 +- .../trees/plans/logical/LogicalGenerate.java |6 +- .../nereids/trees/plans/logical/LogicalJoin.java |6 +- .../trees/plans/logical/LogicalOlapScan.java | 21 +- .../trees/plans/logical/LogicalProject.java| 13 +- .../trees/plans/logical/LogicalSetOperation.java | 30 +- .../nereids/trees/plans/logical/LogicalUnion.java |4 +- .../trees/plans/visitor/InferPlanOutputAlias.java | 68 +- .../org/apache/doris/nereids/types/ArrayType.java |3 +- .../org/apache/doris/nereids/types/MapType.java|3 +- .../org/apache/doris/nereids/types/StructType.java |3 +- .../apache/doris/nereids/util/ExpressionUtils.java | 169 +- .../org/apache/doris/nereids/util/JoinUtils.java |8 +- .../org/apache/doris/nereids/util/PlanUtils.java | 18 - .../doris/nereids/util/TypeCoercionUtils.java | 82 +- .../java/org/apache/doris/nereids/util/Utils.java | 69 - .../apache/doris/regression/RegressionTest.groovy | 11 +- .../doris/regression/util/OutputUtils.groovy |2 +- 52 files changed, 4527 insertions(+), 15165 deletions(-) delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/ComplexDataType.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/analyzer/MappingSlot.java delete mode 100644 fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ExpressionAnalyzer.java - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](memory) Fix LRU cache deleter and memory tracking [doris]
doris-robot commented on PR #32080: URL: https://github.com/apache/doris/pull/32080#issuecomment-1990984866 TPC-H: Total hot run time: 36643 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 5bd29b5b7bef14d59926a7d60e8076df42469885, data reload: false -- Round 1 -- q1 17628 422140984098 q2 2027163 145 145 q3 10562 910 880 880 q4 4648660 720 660 q5 7623281327912791 q6 181 124 123 123 q7 1194835 799 799 q8 9489208120202020 q9 7376649464606460 q10 8126246424932464 q11 417 218 210 210 q12 762 300 289 289 q13 17983 301730163016 q14 280 259 253 253 q15 493 457 449 449 q16 484 396 391 391 q17 958 551 537 537 q18 6706592758745874 q19 1577145114661451 q20 568 267 286 267 q21 7324317432443174 q22 793 297 292 292 Total cold run time: 107199 ms Total hot run time: 36643 ms - Round 2, with runtime_filter_mode=off - q1 4076404940594049 q2 325 225 226 225 q3 2978282128362821 q4 1889154415291529 q5 5215526452685264 q6 191 115 115 115 q7 2249184518721845 q8 3186329533453295 q9 8580859585698569 q10 6112365136283628 q11 533 455 433 433 q12 735 563 548 548 q13 12415 288528482848 q14 274 246 250 246 q15 490 451 445 445 q16 439 425 398 398 q17 1739149114621462 q18 7714719870477047 q19 6827151315381513 q20 1908169517041695 q21 4846456047064560 q22 533 460 441 441 Total cold run time: 73254 ms Total hot run time: 52976 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](planner) remove input slot for aggregate slot which is not materialized [doris]
github-actions[bot] commented on PR #32092: URL: https://github.com/apache/doris/pull/32092#issuecomment-1990984953 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](planner) remove input slot for aggregate slot which is not materialized [doris]
github-actions[bot] commented on PR #32092: URL: https://github.com/apache/doris/pull/32092#issuecomment-1990984998 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org