Re: [PR] [Fix](Nereids) Fix datatype length wrong when string contains chinese [doris]
doris-robot commented on PR #29885: URL: https://github.com/apache/doris/pull/29885#issuecomment-1888608966 TPC-DS: Total hot run time: 178560 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 0a1fd5c3536d37d0a733071b9f0be73b2bed5725, data reload: false query1 941 348 327 327 query2 6714183619141836 query3 6699206 201 201 query4 27810 22083 22086 22083 query5 8802590 551 551 query6 269 178 178 178 query7 4612276 272 272 query8 222 202 198 198 query9 9192273028412730 query10 469 253 237 237 query11 16166 15514 15328 15328 query12 131 71 75 71 query13 1769391 385 385 query14 12683 713371237123 query15 267 194 198 194 query16 6337248 242 242 query17 2006493 476 476 query18 1972283 261 261 query19 292 152 148 148 query20 77 76 70 70 query21 192 127 134 127 query22 5291515951835159 query23 31802 30854 31030 30854 query24 12096 282627522752 query25 546 336 326 326 query26 1716151 149 149 query27 2915270 281 270 query28 7174183218331832 query29 2078726 666 666 query30 275 144 146 144 query31 972 753 768 753 query32 91 59 60 59 query33 712 237 245 237 query34 1106471 473 471 query35 886 791 737 737 query36 1315121911781178 query37 186 68 73 68 query38 3413329633123296 query39 1352127712801277 query40 321 103 94 94 query41 40 34 35 34 query42 102 93 93 93 query43 565 506 512 506 query44 1081693 716 693 query45 215 188 186 186 query46 1055660 667 660 query47 1665157116111571 query48 402 333 324 324 query49 1279320 327 320 query50 686 309 315 309 query51 5344521651215121 query52 101 87 102 87 query53 354 287 297 287 query54 923 464 468 464 query55 92 96 88 88 query56 204 194 190 190 query57 979 959 957 957 query58 210 191 191 191 query59 2656258626652586 query60 240 207 224 207 query61 85 81 90 81 query62 643 398 362 362 query63 298 284 291 284 query64 5687170016751675 query65 3378325232413241 query66 1236321 323 321 query67 15852 15290 15276 15276 query68 11537 498 515 498 query69 691 386 395 386 query70 1672151515181515 query71 593 267 232 232 query72 5145278628172786 query73 2125314 309 309 query74 7028649064416441 query75 5262229823092298 query76 6803111810901090 query77 951 270 284 270 query78 10127 877185488548 query79 1020495 487 487 query80 627 340 348 340 query81 448 213 215 213 query82 208 89 89 89 query83 157 121 128 121 query84 285 75 74 74 query85 1025336 346 336 query86 417 364 368 364 query87 3544344833673367 query88 2933219921782178 query89 430 378 363 363 query90 2056208 197 197 query91 161 134 140 134 query92 59 57 57 57 query93 942 404 427 404 query94 1229181 175 175 query95 527 470 461 461 query96 617 321 321 321 query97 4304415241704152 query98 209 192 184 184 query99 997 696 717 696 Total cold run time: 301370 ms Total hot run time: 178560 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [Fix](Nereids) Fix datatype length wrong when string contains chinese [doris]
doris-robot commented on PR #29885: URL: https://github.com/apache/doris/pull/29885#issuecomment-1888609876 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.4 seconds stream load tsv: 559 seconds loaded 74807831229 Bytes, about 127 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 21.7 seconds inserted 1000 Rows, about 460K ops/s storage size: 17183905532 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipelineX](improvement) Adjust local exchange strategy [doris]
github-actions[bot] commented on PR #29915: URL: https://github.com/apache/doris/pull/29915#issuecomment-1888614446 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Nereids) Fix datatype length wrong when string contains chinese [doris]
doris-robot commented on PR #29885: URL: https://github.com/apache/doris/pull/29885#issuecomment-1888615321 ClickBench: Total hot run time: 30.27 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 0a1fd5c3536d37d0a733071b9f0be73b2bed5725, data reload: false query1 0.060.060.06 query2 0.060.020.03 query3 0.240.120.11 query4 1.780.120.11 query5 0.520.510.53 query6 1.390.630.63 query7 0.020.010.02 query8 0.040.030.03 query9 0.550.500.49 query10 0.540.540.54 query11 0.120.090.09 query12 0.110.090.10 query13 0.610.610.61 query14 0.770.800.77 query15 0.820.800.80 query16 0.340.350.34 query17 0.971.011.00 query18 0.260.250.25 query19 1.851.781.73 query20 0.010.020.02 query21 15.41 0.560.57 query22 2.552.551.74 query23 17.15 0.930.71 query24 16.15 0.620.57 query25 2.160.150.14 query26 0.140.130.13 query27 0.140.140.14 query28 5.620.840.87 query29 12.63 3.243.17 query30 0.560.520.47 query31 2.780.360.35 query32 3.410.480.48 query33 3.173.223.20 query34 15.90 4.194.14 query35 4.144.164.14 query36 1.081.031.04 query37 0.070.050.05 query38 0.040.020.02 query39 0.020.010.02 query40 0.160.130.13 query41 0.070.010.02 query42 0.020.020.01 query43 0.030.020.02 Total cold run time: 114.46 s Total hot run time: 30.27 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](function) support ip function named ipv6_cidr_to_range(addr, cidr) [doris]
amorynan commented on code in PR #29812:
URL: https://github.com/apache/doris/pull/29812#discussion_r1450010271
##
be/src/vec/functions/function_ip.h:
##
@@ -832,4 +835,128 @@ class FunctionIsIPAddressInRange : public IFunction {
}
};
+class FunctionIPv6CIDRToRange : public IFunction {
+public:
+static constexpr auto name = "ipv6_cidr_to_range";
+static FunctionPtr create() { return
std::make_shared(); }
+
+String get_name() const override { return name; }
+
+size_t get_number_of_arguments() const override { return 2; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+const auto& ipv6_type = arguments[0];
+if (!is_string(remove_nullable(ipv6_type)) &&
!is_ipv6(remove_nullable(ipv6_type))) {
+throw Exception(
+ErrorCode::INVALID_ARGUMENT,
+"Illegal type {} of first argument of function {},
expected String or IPv6",
+ipv6_type->get_name(), get_name());
+}
+const auto& cidr_type = arguments[1];
+if (!is_integer(remove_nullable(cidr_type))) {
+throw Exception(ErrorCode::INVALID_ARGUMENT,
+"Illegal type {} of second argument of function
{}, expected Integer",
+cidr_type->get_name(), get_name());
+}
+DataTypePtr element = std::make_shared();
+return make_nullable(std::make_shared(DataTypes
{element, element},
+ Strings {"min",
"max"}));
+}
+
+bool use_default_implementation_for_nulls() const override { return true; }
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) const override
{
+ColumnPtr& addr_column = block.get_by_position(arguments[0]).column;
+ColumnPtr& cidr_column = block.get_by_position(arguments[1]).column;
+const IColumn* cidr_col = nullptr;
+const NullMap* cidr_nullmap = nullptr;
+ColumnPtr col_res = nullptr;
+
+if (cidr_column->is_nullable()) {
+const auto* cidr_column_nullable =
+assert_cast(cidr_column.get());
+cidr_col = &cidr_column_nullable->get_nested_column();
+cidr_nullmap = &cidr_column_nullable->get_null_map_data();
+} else {
+cidr_col = cidr_column.get();
+}
+
+if (addr_column->is_nullable()) {
+const auto* addr_column_nullable =
+assert_cast(addr_column.get());
+const NullMap* addr_nullmap =
&addr_column_nullable->get_null_map_data();
+if (const auto* ipv6_addr_column =
check_and_get_column(
Review Comment:
same with is_null
use data type to check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](function) support ip function named ipv6_cidr_to_range(addr, cidr) [doris]
amorynan commented on code in PR #29812:
URL: https://github.com/apache/doris/pull/29812#discussion_r1450009453
##
be/src/vec/functions/function_ip.h:
##
@@ -832,4 +835,128 @@ class FunctionIsIPAddressInRange : public IFunction {
}
};
+class FunctionIPv6CIDRToRange : public IFunction {
+public:
+static constexpr auto name = "ipv6_cidr_to_range";
+static FunctionPtr create() { return
std::make_shared(); }
+
+String get_name() const override { return name; }
+
+size_t get_number_of_arguments() const override { return 2; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+const auto& ipv6_type = arguments[0];
+if (!is_string(remove_nullable(ipv6_type)) &&
!is_ipv6(remove_nullable(ipv6_type))) {
+throw Exception(
+ErrorCode::INVALID_ARGUMENT,
+"Illegal type {} of first argument of function {},
expected String or IPv6",
+ipv6_type->get_name(), get_name());
+}
+const auto& cidr_type = arguments[1];
+if (!is_integer(remove_nullable(cidr_type))) {
+throw Exception(ErrorCode::INVALID_ARGUMENT,
+"Illegal type {} of second argument of function
{}, expected Integer",
+cidr_type->get_name(), get_name());
+}
+DataTypePtr element = std::make_shared();
+return make_nullable(std::make_shared(DataTypes
{element, element},
+ Strings {"min",
"max"}));
+}
+
+bool use_default_implementation_for_nulls() const override { return true; }
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) const override
{
+ColumnPtr& addr_column = block.get_by_position(arguments[0]).column;
+ColumnPtr& cidr_column = block.get_by_position(arguments[1]).column;
+const IColumn* cidr_col = nullptr;
+const NullMap* cidr_nullmap = nullptr;
+ColumnPtr col_res = nullptr;
+
+if (cidr_column->is_nullable()) {
Review Comment:
do not use columntype to judge what type of current data, use
block.get_data_type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](statistics)support statistics for iceberg/paimon/hudi table [doris]
doris-robot commented on PR #29868: URL: https://github.com/apache/doris/pull/29868#issuecomment-1888617515 TPC-DS: Total hot run time: 177800 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 6f4436d35c474f1286fa9acf5c5f53eef59f272f, data reload: false query1 932 337 329 329 query2 6716194319641943 query3 6724203 204 203 query4 26231 22132 22153 22132 query5 8537571 572 571 query6 282 192 176 176 query7 4611286 278 278 query8 234 197 190 190 query9 9205272028562720 query10 510 234 243 234 query11 16216 15314 15410 15314 query12 139 76 76 76 query13 1696385 374 374 query14 12109 718373067183 query15 260 194 195 194 query16 5802257 242 242 query17 1915479 486 479 query18 1956262 264 262 query19 282 146 147 146 query20 84 77 69 69 query21 192 132 121 121 query22 4960467347434673 query23 31852 30823 30800 30800 query24 11972 280627512751 query25 556 319 327 319 query26 1772147 156 147 query27 2819277 276 276 query28 7033186518471847 query29 2086677 653 653 query30 280 145 142 142 query31 979 760 762 760 query32 96 61 60 60 query33 717 243 251 243 query34 1068465 459 459 query35 888 729 723 723 query36 1343125311861186 query37 181 71 67 67 query38 3443325632963256 query39 1348128112781278 query40 323 93 94 93 query41 40 35 34 34 query42 107 95 100 95 query43 558 506 523 506 query44 1108697 708 697 query45 209 181 185 181 query46 1060655 651 651 query47 1672157515581558 query48 406 330 333 330 query49 1281315 323 315 query50 696 323 314 314 query51 5356518151605160 query52 108 99 89 89 query53 369 287 286 286 query54 910 449 454 449 query55 87 86 89 86 query56 219 192 197 192 query57 980 947 943 943 query58 201 198 194 194 query59 2726261824952495 query60 237 214 214 214 query61 83 80 81 80 query62 635 403 393 393 query63 293 284 282 282 query64 5768167716541654 query65 3325324632443244 query66 1267334 327 327 query67 15638 15301 15257 15257 query68 12002 516 520 516 query69 691 380 377 377 query70 1685146815371468 query71 570 256 243 243 query72 4884281928222819 query73 2438316 309 309 query74 6980637564426375 query75 4988227823502278 query76 6337920 1098920 query77 768 269 261 261 query78 9914850784978497 query79 1024492 492 492 query80 552 335 343 335 query81 457 206 205 205 query82 215 95 90 90 query83 151 121 121 121 query84 278 73 74 73 query85 1061345 343 343 query86 396 364 418 364 query87 3602340733793379 query88 3050219722122197 query89 455 386 393 386 query90 1940200 205 200 query91 167 125 134 125 query92 64 56 53 53 query93 971 407 403 403 query94 1212187 179 179 query95 522 485 481 481 query96 627 316 316 316 query97 4293413041574130 query98 193 193 183 183 query99 1022687 723 687 Total cold run time: 297197 ms Total hot run time: 177800 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s
Re: [PR] [feature](statistics)support statistics for iceberg/paimon/hudi table [doris]
doris-robot commented on PR #29868: URL: https://github.com/apache/doris/pull/29868#issuecomment-1888617956 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.91 seconds stream load tsv: 559 seconds loaded 74807831229 Bytes, about 127 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 22.0 seconds inserted 1000 Rows, about 454K ops/s storage size: 17183905479 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Nereids) Fix datatype length wrong when string contains chinese [doris]
doris-robot commented on PR #29885: URL: https://github.com/apache/doris/pull/29885#issuecomment-1888618918 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 0a1fd5c3536d37d0a733071b9f0be73b2bed5725 with default session variables Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s Insert into select: 13.4 seconds inserted 1000 Rows, about 746K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [deps](hadoop) upgrade hadoop deps to 3.3.4.6 #29908 [doris]
doris-robot commented on PR #29918: URL: https://github.com/apache/doris/pull/29918#issuecomment-1888619929 TeamCity be ut coverage result: Function Coverage: 37.84% (8026/21208) Line Coverage: 29.51% (65292/221280) Region Coverage: 28.96% (33588/115983) Branch Coverage: 24.81% (17234/69456) Coverage Report: http://coverage.selectdb-in.cc/coverage/49c15136e2abd3ae5008e62f6582bdbf38a10f76_49c15136e2abd3ae5008e62f6582bdbf38a10f76/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](serde)update slice reserve and deduce slice back usage [doris]
amorynan commented on PR #29879: URL: https://github.com/apache/doris/pull/29879#issuecomment-1888623130 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](statistics)support statistics for iceberg/paimon/hudi table [doris]
doris-robot commented on PR #29868: URL: https://github.com/apache/doris/pull/29868#issuecomment-1888629142 ClickBench: Total hot run time: 30.05 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 6f4436d35c474f1286fa9acf5c5f53eef59f272f, data reload: false query1 0.050.060.05 query2 0.060.030.03 query3 0.250.110.10 query4 1.790.120.12 query5 0.520.530.51 query6 1.350.650.63 query7 0.020.020.01 query8 0.040.030.03 query9 0.550.490.49 query10 0.540.560.56 query11 0.120.090.10 query12 0.120.100.09 query13 0.610.600.60 query14 0.770.800.78 query15 0.820.800.81 query16 0.350.350.36 query17 0.960.931.00 query18 0.260.250.27 query19 1.861.751.77 query20 0.010.000.01 query21 15.40 0.560.59 query22 2.452.201.45 query23 17.49 0.870.82 query24 16.02 0.580.60 query25 2.160.160.15 query26 0.130.140.13 query27 0.140.150.15 query28 6.050.820.81 query29 12.55 3.153.20 query30 0.500.460.47 query31 2.790.370.35 query32 3.390.490.48 query33 3.203.243.22 query34 15.83 4.154.15 query35 4.164.124.15 query36 1.101.041.04 query37 0.060.050.05 query38 0.040.030.03 query39 0.020.020.01 query40 0.160.130.13 query41 0.070.010.02 query42 0.020.020.02 query43 0.020.010.02 Total cold run time: 114.8 s Total hot run time: 30.05 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](statistics)support statistics for iceberg/paimon/hudi table [doris]
doris-robot commented on PR #29868: URL: https://github.com/apache/doris/pull/29868#issuecomment-1888633015 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 6f4436d35c474f1286fa9acf5c5f53eef59f272f with default session variables Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s Insert into select: 12.2 seconds inserted 1000 Rows, about 819K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipelineX](improvement) Adjust local exchange strategy [doris]
doris-robot commented on PR #29915: URL: https://github.com/apache/doris/pull/29915#issuecomment-1888633013 TPC-H: Total hot run time: 39186 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 1c9d5a5647c5621ec9891a1742bd823cfc647327, data reload: false -- Round 1 -- q1 17908 501749734973 q2 2041136 138 136 q3 10687 110411611104 q4 10514 780 821 780 q5 7956324732523247 q6 211 129 127 127 q7 909 516 507 507 q8 9767203520572035 q9 7844665165956595 q10 8269312031063106 q11 426 213 205 205 q12 366 191 192 191 q13 18091 343934393439 q14 248 216 216 216 q15 552 514 511 511 q16 431 377 377 377 q17 948 566 518 518 q18 7342685467536753 q19 1608143914041404 q20 475 293 274 274 q21 2817238924272389 q22 344 303 299 299 Total cold run time: 109754 ms Total hot run time: 39186 ms - Round 2, with runtime_filter_mode=off - q1 5015495549004900 q2 311 200 201 200 q3 3341332933143314 q4 2243226022642260 q5 5824580658065806 q6 191 119 119 119 q7 2305184418731844 q8 3471356335893563 q9 8865880788128807 q10 3743382438263824 q11 553 414 429 414 q12 805 608 599 599 q13 3882321631993199 q14 293 254 265 254 q15 570 522 517 517 q16 513 473 456 456 q17 2057203020112011 q18 8729831983808319 q19 1616163716121612 q20 2156194519161916 q21 6072573956895689 q22 546 475 443 443 Total cold run time: 63101 ms Total hot run time: 60066 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud rowset meta & tablet meta definition [doris]
doris-robot commented on PR #29911: URL: https://github.com/apache/doris/pull/29911#issuecomment-1888634149 TPC-H: Total hot run time: 38525 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 0b88f15504a17501c7cf0b7e969bea5e4679fecd, data reload: false -- Round 1 -- q1 17977 499249994992 q2 2046144 128 128 q3 10695 114312031143 q4 10244 813 809 809 q5 7807317431023102 q6 205 128 125 125 q7 872 505 483 483 q8 9260196219971962 q9 7334645264576452 q10 8238309530703070 q11 420 204 213 204 q12 365 194 194 194 q13 18091 343633883388 q14 243 210 211 210 q15 561 514 503 503 q16 427 380 376 376 q17 950 493 464 464 q18 7275667166616661 q19 1608129614701296 q20 589 297 307 297 q21 2840236024262360 q22 349 317 306 306 Total cold run time: 108396 ms Total hot run time: 38525 ms - Round 2, with runtime_filter_mode=off - q1 5020492949944929 q2 316 200 198 198 q3 3312332532943294 q4 2238224122552241 q5 5772575757855757 q6 195 120 121 120 q7 2310183018751830 q8 3433355335763553 q9 8801882288188818 q10 3742382438143814 q11 541 423 420 420 q12 771 605 601 601 q13 6165319632093196 q14 280 259 267 259 q15 563 515 506 506 q16 495 452 459 452 q17 2023204919891989 q18 8760840882138213 q19 1623161616171616 q20 2138193319241924 q21 6043576358315763 q22 530 474 473 473 Total cold run time: 65071 ms Total hot run time: 59966 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](function) support ip function named ipv6_cidr_to_range(addr, cidr) [doris]
amorynan commented on code in PR #29812:
URL: https://github.com/apache/doris/pull/29812#discussion_r1450024745
##
be/src/vec/functions/function_ip.h:
##
@@ -832,4 +835,128 @@ class FunctionIsIPAddressInRange : public IFunction {
}
};
+class FunctionIPv6CIDRToRange : public IFunction {
+public:
+static constexpr auto name = "ipv6_cidr_to_range";
+static FunctionPtr create() { return
std::make_shared(); }
+
+String get_name() const override { return name; }
+
+size_t get_number_of_arguments() const override { return 2; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+const auto& ipv6_type = arguments[0];
+if (!is_string(remove_nullable(ipv6_type)) &&
!is_ipv6(remove_nullable(ipv6_type))) {
+throw Exception(
+ErrorCode::INVALID_ARGUMENT,
+"Illegal type {} of first argument of function {},
expected String or IPv6",
+ipv6_type->get_name(), get_name());
+}
+const auto& cidr_type = arguments[1];
+if (!is_integer(remove_nullable(cidr_type))) {
+throw Exception(ErrorCode::INVALID_ARGUMENT,
+"Illegal type {} of second argument of function
{}, expected Integer",
+cidr_type->get_name(), get_name());
+}
+DataTypePtr element = std::make_shared();
+return make_nullable(std::make_shared(DataTypes
{element, element},
+ Strings {"min",
"max"}));
+}
+
+bool use_default_implementation_for_nulls() const override { return true; }
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) const override
{
+ColumnPtr& addr_column = block.get_by_position(arguments[0]).column;
+ColumnPtr& cidr_column = block.get_by_position(arguments[1]).column;
+const IColumn* cidr_col = nullptr;
+const NullMap* cidr_nullmap = nullptr;
+ColumnPtr col_res = nullptr;
+
+if (cidr_column->is_nullable()) {
+const auto* cidr_column_nullable =
+assert_cast(cidr_column.get());
+cidr_col = &cidr_column_nullable->get_nested_column();
+cidr_nullmap = &cidr_column_nullable->get_null_map_data();
+} else {
+cidr_col = cidr_column.get();
Review Comment:
if cidr column is column vector , just assert cast. do not use IColumn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipelineX](improvement) Adjust local exchange strategy [doris]
doris-robot commented on PR #29915: URL: https://github.com/apache/doris/pull/29915#issuecomment-1888637221 TeamCity be ut coverage result: Function Coverage: 36.71% (8650/23566) Line Coverage: 28.72% (70444/245294) Region Coverage: 27.63% (36397/131722) Branch Coverage: 24.35% (18623/76480) Coverage Report: http://coverage.selectdb-in.cc/coverage/1c9d5a5647c5621ec9891a1742bd823cfc647327_1c9d5a5647c5621ec9891a1742bd823cfc647327/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](function) support ip function named ipv6_cidr_to_range(addr, cidr) [doris]
amorynan commented on code in PR #29812:
URL: https://github.com/apache/doris/pull/29812#discussion_r1450025323
##
be/src/vec/functions/function_ip.h:
##
@@ -832,4 +835,128 @@ class FunctionIsIPAddressInRange : public IFunction {
}
};
+class FunctionIPv6CIDRToRange : public IFunction {
+public:
+static constexpr auto name = "ipv6_cidr_to_range";
+static FunctionPtr create() { return
std::make_shared(); }
+
+String get_name() const override { return name; }
+
+size_t get_number_of_arguments() const override { return 2; }
+
+DataTypePtr get_return_type_impl(const DataTypes& arguments) const
override {
+const auto& ipv6_type = arguments[0];
+if (!is_string(remove_nullable(ipv6_type)) &&
!is_ipv6(remove_nullable(ipv6_type))) {
+throw Exception(
+ErrorCode::INVALID_ARGUMENT,
+"Illegal type {} of first argument of function {},
expected String or IPv6",
+ipv6_type->get_name(), get_name());
+}
+const auto& cidr_type = arguments[1];
+if (!is_integer(remove_nullable(cidr_type))) {
+throw Exception(ErrorCode::INVALID_ARGUMENT,
+"Illegal type {} of second argument of function
{}, expected Integer",
+cidr_type->get_name(), get_name());
+}
+DataTypePtr element = std::make_shared();
+return make_nullable(std::make_shared(DataTypes
{element, element},
+ Strings {"min",
"max"}));
+}
+
+bool use_default_implementation_for_nulls() const override { return true; }
+
+Status execute_impl(FunctionContext* context, Block& block, const
ColumnNumbers& arguments,
+size_t result, size_t input_rows_count) const override
{
+ColumnPtr& addr_column = block.get_by_position(arguments[0]).column;
+ColumnPtr& cidr_column = block.get_by_position(arguments[1]).column;
+const IColumn* cidr_col = nullptr;
+const NullMap* cidr_nullmap = nullptr;
+ColumnPtr col_res = nullptr;
+
+if (cidr_column->is_nullable()) {
+const auto* cidr_column_nullable =
+assert_cast(cidr_column.get());
+cidr_col = &cidr_column_nullable->get_nested_column();
+cidr_nullmap = &cidr_column_nullable->get_null_map_data();
+} else {
+cidr_col = cidr_column.get();
+}
+
+if (addr_column->is_nullable()) {
+const auto* addr_column_nullable =
+assert_cast(addr_column.get());
+const NullMap* addr_nullmap =
&addr_column_nullable->get_null_map_data();
+if (const auto* ipv6_addr_column =
check_and_get_column(
+addr_column_nullable->get_nested_column())) {
+col_res = execute_impl(*ipv6_addr_column,
addr_nullmap, *cidr_col,
+ cidr_nullmap,
input_rows_count);
+} else if (const auto* str_addr_column =
check_and_get_column(
+ addr_column_nullable->get_nested_column())) {
+col_res = execute_impl(*str_addr_column,
addr_nullmap, *cidr_col,
+ cidr_nullmap,
input_rows_count);
+} else {
+return Status::RuntimeError("Illegal column {} of argument of
function {}",
+addr_column->get_name(),
get_name());
+}
+} else {
+if (const auto* ipv6_addr_column =
+check_and_get_column(addr_column.get())) {
+col_res = execute_impl(*ipv6_addr_column, nullptr,
*cidr_col, nullptr,
+ input_rows_count);
+} else if (const auto* str_addr_column =
+
check_and_get_column(addr_column.get())) {
+col_res = execute_impl(*str_addr_column,
nullptr, *cidr_col, nullptr,
+ input_rows_count);
+} else {
+return Status::RuntimeError("Illegal column {} of argument of
function {}",
+addr_column->get_name(),
get_name());
+}
+}
+
+block.replace_by_position(result, std::move(col_res));
+return Status::OK();
+}
+
+template
+static ColumnPtr execute_impl(const FromColumn& from_column, const
NullMap* from_nullmap,
+ const IColumn& cidr_column, const NullMap*
cidr_nullmap,
+ size_t input_rows_count) {
+auto col_res_lower_range = ColumnIPv6::create(input_rows_count, 0);
+auto col_res_upper_range = ColumnIPv6::create(input_rows_count, 0);
+auto col_res_null_map = ColumnUInt8::create(
Re: [PR] [release](version) change branch-2.0 version to doris-2.0.4 [doris]
doris-robot commented on PR #29917: URL: https://github.com/apache/doris/pull/29917#issuecomment-1888638920 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 49.81 seconds stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.8 seconds inserted 1000 Rows, about 335K ops/s storage size: 17167739722 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [bug](bitmap) should return error status when execute failed (#29841)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 13349fe355a [bug](bitmap) should return error status when execute
failed (#29841)
13349fe355a is described below
commit 13349fe355a4765acde7eeb27615b9f4c79bb209
Author: zhangstar333 <87313068+zhangstar...@users.noreply.github.com>
AuthorDate: Fri Jan 12 16:22:14 2024 +0800
[bug](bitmap) should return error status when execute failed (#29841)
---
be/src/vec/functions/function_bitmap.cpp | 14 +++---
be/src/vec/functions/function_bitmap_variadic.cpp | 4 ++--
be/src/vec/functions/function_coalesce.cpp | 2 +-
be/src/vec/functions/function_encryption.cpp | 5 ++---
be/src/vec/functions/function_hex.cpp | 2 +-
be/src/vec/functions/function_ifnull.h | 3 +--
be/src/vec/functions/function_quantile_state.cpp | 6 ++
be/src/vec/functions/function_reverse.h| 5 ++---
be/src/vec/functions/function_string.cpp | 11 ++-
be/src/vec/functions/function_string.h | 8
be/src/vec/functions/like.cpp | 2 +-
.../query_p0/sql_functions/case_function/case_null1.csv| 4 ++--
12 files changed, 31 insertions(+), 35 deletions(-)
diff --git a/be/src/vec/functions/function_bitmap.cpp
b/be/src/vec/functions/function_bitmap.cpp
index 08b15098755..88147ec6cd9 100644
--- a/be/src/vec/functions/function_bitmap.cpp
+++ b/be/src/vec/functions/function_bitmap.cpp
@@ -361,7 +361,7 @@ public:
const auto& str_column = static_cast(*argument_column);
const ColumnString::Chars& data = str_column.get_chars();
const ColumnString::Offsets& offsets = str_column.get_offsets();
-static_cast(Impl::vector(data, offsets, res, null_map,
input_rows_count));
+RETURN_IF_ERROR(Impl::vector(data, offsets, res, null_map,
input_rows_count));
} else if constexpr (std::is_same_v) {
auto argument_type = remove_nullable(
assert_cast(*block.get_by_position(arguments[0]).type)
@@ -375,19 +375,19 @@ public:
WhichDataType which_type(argument_type);
if (which_type.is_int8()) {
-static_cast(Impl::template vector(
-offset_column_data, nested_column, nested_null_map,
res, null_map));
+RETURN_IF_ERROR(Impl::template
vector(offset_column_data, nested_column,
+
nested_null_map, res, null_map));
} else if (which_type.is_uint8()) {
-static_cast(Impl::template vector(
+RETURN_IF_ERROR(Impl::template vector(
offset_column_data, nested_column, nested_null_map,
res, null_map));
} else if (which_type.is_int16()) {
-static_cast(Impl::template vector(
+RETURN_IF_ERROR(Impl::template vector(
offset_column_data, nested_column, nested_null_map,
res, null_map));
} else if (which_type.is_int32()) {
-static_cast(Impl::template vector(
+RETURN_IF_ERROR(Impl::template vector(
offset_column_data, nested_column, nested_null_map,
res, null_map));
} else if (which_type.is_int64()) {
-static_cast(Impl::template vector(
+RETURN_IF_ERROR(Impl::template vector(
offset_column_data, nested_column, nested_null_map,
res, null_map));
} else {
return Status::RuntimeError("Illegal column {} of argument of
function {}",
diff --git a/be/src/vec/functions/function_bitmap_variadic.cpp
b/be/src/vec/functions/function_bitmap_variadic.cpp
index 321da1a0601..9fd22a1a052 100644
--- a/be/src/vec/functions/function_bitmap_variadic.cpp
+++ b/be/src/vec/functions/function_bitmap_variadic.cpp
@@ -250,8 +250,8 @@ public:
auto& vec_res = col_res->get_data();
vec_res.resize(input_rows_count);
-static_cast(Impl::vector_vector(argument_columns, argument_size,
input_rows_count,
- vec_res, col_res_nulls));
+RETURN_IF_ERROR(Impl::vector_vector(argument_columns, argument_size,
input_rows_count,
+vec_res, col_res_nulls));
if (!use_default_implementation_for_nulls() &&
result_info.type->is_nullable()) {
block.replace_by_position(
result, ColumnNullable::create(std::move(col_res),
std::move(col_res_nulls)));
diff --git a/be/src/vec/functions/function_coalesce.cpp
b/be/src/vec/functions/function_coalesce.cpp
in
Re: [PR] [bug](bitmap) should return error status when execute failed [doris]
yiguolei merged PR #29841: URL: https://github.com/apache/doris/pull/29841 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud rowset meta & tablet meta definition [doris]
doris-robot commented on PR #29911: URL: https://github.com/apache/doris/pull/29911#issuecomment-1888644224 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 47.02 seconds stream load tsv: 561 seconds loaded 74807831229 Bytes, about 127 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s insert into select: 21.6 seconds inserted 1000 Rows, about 462K ops/s storage size: 17183830923 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](merge-cloud) Decouple DataDir from global StorageEngine instance [doris]
github-actions[bot] commented on PR #29849: URL: https://github.com/apache/doris/pull/29849#issuecomment-1888647889 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipelineX](improvement) Adjust local exchange strategy [doris]
doris-robot commented on PR #29915: URL: https://github.com/apache/doris/pull/29915#issuecomment-1888647968 TPC-DS: Total hot run time: 178018 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 1c9d5a5647c5621ec9891a1742bd823cfc647327, data reload: false query1 925 334 329 329 query2 6719188518201820 query3 6723212 209 209 query4 26220 22168 22098 22098 query5 8183535 586 535 query6 276 185 197 185 query7 4619283 267 267 query8 243 206 200 200 query9 9219279227672767 query10 650 250 252 250 query11 16066 15376 15348 15348 query12 126 76 73 73 query13 1699396 394 394 query14 11762 712871987128 query15 248 189 196 189 query16 5288259 245 245 query17 1869482 481 481 query18 1955261 260 260 query19 285 146 149 146 query20 80 79 70 70 query21 191 137 125 125 query22 5073481147304730 query23 31828 30801 30738 30738 query24 12284 281327992799 query25 559 328 321 321 query26 1733146 148 146 query27 2975285 279 279 query28 7304191418921892 query29 2104703 651 651 query30 288 143 140 140 query31 977 752 777 752 query32 92 61 60 60 query33 726 236 240 236 query34 1157469 470 469 query35 897 739 763 739 query36 1367124912621249 query37 186 69 70 69 query38 3445326132993261 query39 1378128712801280 query40 337 95 100 95 query41 37 34 34 34 query42 96 97 95 95 query43 536 516 506 506 query44 1061708 717 708 query45 210 186 177 177 query46 1045673 638 638 query47 1623155515261526 query48 407 330 331 330 query49 1268322 302 302 query50 701 321 314 314 query51 5359517652405176 query52 107 92 99 92 query53 355 287 276 276 query54 902 456 475 456 query55 91 85 87 85 query56 213 197 193 193 query57 958 954 931 931 query58 212 196 197 196 query59 2747250125762501 query60 244 216 214 214 query61 86 84 86 84 query62 646 392 387 387 query63 317 284 290 284 query64 5828168516781678 query65 3340325832663258 query66 1357330 319 319 query67 15570 15179 15154 15154 query68 12767 503 493 493 query69 688 371 374 371 query70 1675149715051497 query71 554 243 241 241 query72 4865285328512851 query73 2972312 318 312 query74 7113644064806440 query75 5345233722782278 query76 6374107410771074 query77 670 280 256 256 query78 9886867685818581 query79 1034500 485 485 query80 565 333 342 333 query81 453 205 209 205 query82 206 87 89 87 query83 136 117 129 117 query84 280 75 73 73 query85 1019345 337 337 query86 368 368 360 360 query87 3607340533773377 query88 3119218521982185 query89 463 377 394 377 query90 1861215 202 202 query91 174 132 127 127 query92 63 57 51 51 query93 1332430 399 399 query94 1226192 183 183 query95 510 476 467 467 query96 610 309 316 309 query97 4261413141334131 query98 204 191 190 190 query99 982 690 694 690 Total cold run time: 298754 ms Total hot run time: 178018 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [feature](merge-cloud) Decouple DataDir from global StorageEngine instance [doris]
github-actions[bot] commented on PR #29849: URL: https://github.com/apache/doris/pull/29849#issuecomment-1888647835 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](aggregate) test [doris]
BiteThet commented on PR #29919: URL: https://github.com/apache/doris/pull/29919#issuecomment-1888648282 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Improvement](aggregate) test [doris]
BiteThet opened a new pull request, #29919: URL: https://github.com/apache/doris/pull/29919 ## Proposed changes test ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](merge-cloud) Decouple DataDir from global StorageEngine instance [doris]
dataroaring merged PR #29849: URL: https://github.com/apache/doris/pull/29849 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (13349fe355a -> 57d9e89390a)
This is an automated email from the ASF dual-hosted git repository. dataroaring pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 13349fe355a [bug](bitmap) should return error status when execute failed (#29841) add 57d9e89390a [feature](merge-cloud) Decouple DataDir from global StorageEngine instance (#29849) No new revisions were added by this update. Summary of changes: be/src/olap/data_dir.cpp | 179 +++-- be/src/olap/data_dir.h | 26 ++-- be/src/olap/olap_server.cpp| 14 +- be/src/olap/snapshot_manager.cpp | 2 +- be/src/olap/storage_engine.cpp | 66 +++-- be/src/olap/storage_engine.h | 10 +- be/src/olap/tablet_manager.cpp | 1 - be/src/olap/tablet_meta.cpp| 18 +-- be/src/olap/tablet_meta.h | 4 +- be/test/olap/delete_bitmap_calculator_test.cpp | 2 +- be/test/olap/ordered_data_compaction_test.cpp | 2 +- be/test/olap/path_gc_test.cpp | 2 +- be/test/olap/rowset/beta_rowset_test.cpp | 5 - be/test/olap/storage_engine_test.cpp | 8 +- be/test/olap/tablet_meta_manager_test.cpp | 6 +- be/test/olap/tablet_mgr_test.cpp | 2 +- be/test/olap/tablet_test.cpp | 5 +- be/test/vec/olap/vertical_compaction_test.cpp | 6 +- 18 files changed, 138 insertions(+), 220 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [Fix](nereids) Fix cte rewrite by mv failure and predicates compensation by mistake (#29820)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/doris.git The following commit(s) were added to refs/heads/master by this push: new 99ad5a83482 [Fix](nereids) Fix cte rewrite by mv failure and predicates compensation by mistake (#29820) 99ad5a83482 is described below commit 99ad5a8348274e41fa136ff80ce1c332dd74c99f Author: seawinde <149132972+seawi...@users.noreply.github.com> AuthorDate: Fri Jan 12 16:30:35 2024 +0800 [Fix](nereids) Fix cte rewrite by mv failure and predicates compensation by mistake (#29820) Fix cte rewrite by mv wrongly when query has scalar aggregate but view no For example as following, it should not be rewritten by materialized view successfully // materialzied view define def mv20_1 = """ select l_shipmode, l_shipinstruct, sum(l_extendedprice), count() from lineitem left join orders on lineitem.L_ORDERKEY = orders.O_ORDERKEY group by l_shipmode, l_shipinstruct; """ // query sql def query20_1 = """ select sum(l_extendedprice), count() from lineitem left join orders on lineitem.L_ORDERKEY = orders.O_ORDERKEY """ Fix predicates compensation by mistake For example as following, it can return right result, but it's wrong earlier. // materialzied view define def mv7_1 = """ select l_shipdate, o_orderdate, l_partkey, l_suppkey from lineitem left join orders on lineitem.l_orderkey = orders.o_orderkey where l_shipdate = '2023-12-08' and o_orderdate = '2023-12-08'; """ // query sql def query7_1 = """ select l_shipdate, o_orderdate, l_partkey, l_suppkey from (select * from lineitem where l_shipdate = '2023-10-17' ) t1 left join orders on t1.l_orderkey = orders.o_orderkey; """ and optimize some code usage and add more comment for method --- .../query-async-materialized-view.md | 1 - .../query-async-materialized-view.md | 1 - .../mv/AbstractMaterializedViewAggregateRule.java | 28 ++- .../mv/AbstractMaterializedViewRule.java | 215 ++--- .../mv/LogicalCompatibilityContext.java| 8 +- .../exploration/mv/MaterializedViewUtils.java | 18 ++ .../nereids/rules/exploration/mv/Predicates.java | 178 +++-- .../nereids/rules/exploration/mv/StructInfo.java | 32 ++- .../plans/visitor/ExpressionLineageReplacer.java | 25 ++- .../joinorder/hypergraph/CompareOuterJoinTest.java | 6 +- .../jobs/joinorder/hypergraph/InferJoinTest.java | 6 +- .../joinorder/hypergraph/InferPredicateTest.java | 6 +- .../joinorder/hypergraph/PullupExpressionTest.java | 6 +- .../exploration/mv/HyperGraphComparatorTest.java | 4 +- .../mv/agg_with_roll_up/aggregate_with_roll_up.out | 36 .../aggregate_without_roll_up.out | 28 +++ .../nereids_rules_p0/mv/join/inner/inner_join.out | 16 ++ .../mv/join/left_outer/outer_join.out | 44 + .../agg_with_roll_up/aggregate_with_roll_up.groovy | 90 +++-- .../aggregate_without_roll_up.groovy | 107 +- .../mv/join/inner/inner_join.groovy| 35 ++-- .../mv/join/left_outer/outer_join.groovy | 82 ++-- 22 files changed, 716 insertions(+), 256 deletions(-) diff --git a/docs/en/docs/query-acceleration/async-materialized-view/query-async-materialized-view.md b/docs/en/docs/query-acceleration/async-materialized-view/query-async-materialized-view.md index 4c6418e7606..8cd13b4e43a 100644 --- a/docs/en/docs/query-acceleration/async-materialized-view/query-async-materialized-view.md +++ b/docs/en/docs/query-acceleration/async-materialized-view/query-async-materialized-view.md @@ -398,7 +398,6 @@ If you want to know the detailed information about materialized view candidates, | SET enable_nereids_planner = true;| Asynchronous materialized views are only supported under the new optimizer, so the new optimizer needs to be enabled. | | SET enable_materialized_view_rewrite = true; | Enable or disable query transparent rewriting, default is disabled | | SET materialized_view_rewrite_enable_contain_external_table = true; | Whether materialized views participating in transparent rewriting are allowed to contain external tables, default is not allowed | -| SET disable_nereids_rules = 'ELIMINATE_OUTER_JOIN'; | Currently, outer join elimination has an impact on transparent | ## Limitations diff --git a/docs/zh-CN/docs/query-acceleration/async-materialized-view/query-async-materialized-view.md b/docs/zh-CN/docs/query-acceleration/async-materialized-view/query-async-materialized-view.md index d35a5383dc8..1921498f
Re: [PR] [pipelineX](improvement) Adjust local exchange strategy [doris]
doris-robot commented on PR #29915: URL: https://github.com/apache/doris/pull/29915#issuecomment-1888649039 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.52 seconds stream load tsv: 558 seconds loaded 74807831229 Bytes, about 127 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 21.0 seconds inserted 1000 Rows, about 476K ops/s storage size: 17183963553 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](serde)update slice reserve and deduce slice back usage [doris]
doris-robot commented on PR #29879: URL: https://github.com/apache/doris/pull/29879#issuecomment-1888649590 TPC-H: Total hot run time: 39222 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit aa04f7a1e74623d30588fcad3ccaf8a149af9190, data reload: false -- Round 1 -- q1 17916 501949954995 q2 2037142 135 135 q3 10695 112711391127 q4 10384 871 831 831 q5 8055328132503250 q6 209 130 132 130 q7 915 530 517 517 q8 9705204620462046 q9 7950665365826582 q10 8264318330943094 q11 422 206 223 206 q12 361 200 201 200 q13 18112 345034293429 q14 245 217 215 215 q15 560 519 507 507 q16 429 387 379 379 q17 955 555 500 500 q18 7348681366566656 q19 1593139414541394 q20 489 309 279 279 q21 2803243824542438 q22 363 316 312 312 Total cold run time: 109810 ms Total hot run time: 39222 ms - Round 2, with runtime_filter_mode=off - q1 5005496449594959 q2 310 198 200 198 q3 3342333633203320 q4 2250224922802249 q5 5811582458305824 q6 197 122 123 122 q7 2324186318571857 q8 3464354535703545 q9 8912886088428842 q10 3765383838483838 q11 550 422 423 422 q12 783 594 611 594 q13 3985325532233223 q14 286 263 266 263 q15 562 512 509 509 q16 501 464 446 446 q17 2049204420482044 q18 8775833883118311 q19 1631161716131613 q20 2192197719401940 q21 6075573257135713 q22 552 445 481 445 Total cold run time: 63321 ms Total hot run time: 60277 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud rowset meta & tablet meta definition [doris]
doris-robot commented on PR #29911: URL: https://github.com/apache/doris/pull/29911#issuecomment-1888649044 TPC-DS: Total hot run time: 178488 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 0b88f15504a17501c7cf0b7e969bea5e4679fecd, data reload: false query1 926 339 320 320 query2 6730193818381838 query3 6703211 204 204 query4 25865 21957 22192 21957 query5 8778577 560 560 query6 281 175 208 175 query7 4635273 269 269 query8 233 216 195 195 query9 9171279927462746 query10 584 242 243 242 query11 16174 15395 15247 15247 query12 138 70 71 70 query13 1761394 388 388 query14 12481 717571717171 query15 276 179 186 179 query16 6237266 253 253 query17 1898497 482 482 query18 1972262 259 259 query19 280 144 150 144 query20 80 79 73 73 query21 192 130 122 122 query22 5211493750274937 query23 31690 30946 30818 30818 query24 11858 283028322830 query25 562 331 330 330 query26 1708145 147 145 query27 2828277 281 277 query28 6929184218381838 query29 2079679 681 679 query30 280 134 145 134 query31 974 772 780 772 query32 94 64 57 57 query33 717 247 239 239 query34 1027460 451 451 query35 903 785 774 774 query36 1334135212541254 query37 108 65 68 65 query38 3463330033103300 query39 1341126612891266 query40 319 98 93 93 query41 40 35 34 34 query42 102 90 96 90 query43 570 504 506 504 query44 1089700 700 700 query45 208 183 195 183 query46 1053656 655 655 query47 1677157115891571 query48 413 320 327 320 query49 1271307 311 307 query50 693 315 322 315 query51 5345525252275227 query52 99 95 83 83 query53 361 287 285 285 query54 914 458 456 456 query55 87 92 90 90 query56 219 192 191 191 query57 1012970 936 936 query58 218 193 188 188 query59 2746256623942394 query60 225 229 222 222 query61 85 90 83 83 query62 632 402 392 392 query63 307 273 282 273 query64 5740169117051691 query65 3340327332753273 query66 1220339 326 326 query67 15631 15311 15211 15211 query68 12366 491 514 491 query69 692 379 396 379 query70 1737161516041604 query71 556 235 251 235 query72 5097286528842865 query73 2529311 310 310 query74 6990636564366365 query75 5232234522932293 query76 6401109811151098 query77 905 271 291 271 query78 9972860485858585 query79 1057496 496 496 query80 610 353 347 347 query81 454 208 209 208 query82 212 87 85 85 query83 168 124 123 123 query84 286 77 75 75 query85 1057346 336 336 query86 408 396 398 396 query87 3588339033673367 query88 3186219622112196 query89 475 386 382 382 query90 1920206 206 206 query91 164 130 129 129 query92 67 54 55 54 query93 1973410 406 406 query94 1278193 185 185 query95 514 487 478 478 query96 638 316 313 313 query97 4258412441744124 query98 207 193 183 183 query99 1006757 750 750 Total cold run time: 300120 ms Total hot run time: 178488 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the
Re: [PR] [Fix](nereids) Fix cte rewrite by mv failure and predicates compensation by mistake [doris]
yiguolei merged PR #29820: URL: https://github.com/apache/doris/pull/29820 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Export) Fix the NPE exception when cancel an export job [doris]
yiguolei merged PR #29913: URL: https://github.com/apache/doris/pull/29913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](serde)update slice reserve and deduce slice back usage [doris]
doris-robot commented on PR #29879: URL: https://github.com/apache/doris/pull/29879#issuecomment-1888650534 TeamCity be ut coverage result: Function Coverage: 36.71% (8650/23565) Line Coverage: 28.72% (70454/245285) Region Coverage: 27.64% (36401/131715) Branch Coverage: 24.36% (18626/76476) Coverage Report: http://coverage.selectdb-in.cc/coverage/aa04f7a1e74623d30588fcad3ccaf8a149af9190_aa04f7a1e74623d30588fcad3ccaf8a149af9190/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [deps](hadoop) upgrade hadoop deps to 3.3.4.6 #29908 [doris]
doris-robot commented on PR #29918: URL: https://github.com/apache/doris/pull/29918#issuecomment-1888650628 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 48.96 seconds stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 29.5 seconds inserted 1000 Rows, about 338K ops/s storage size: 17167536214 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [Fix](Export) Fix the NPE exception when cancel an export job #29913
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 532766764ea [Fix](Export) Fix the NPE exception when cancel an export
job #29913
532766764ea is described below
commit 532766764ea1b802cb620988cb89b4ce3bc47699
Author: Tiewei Fang <43782773+bepppo...@users.noreply.github.com>
AuthorDate: Fri Jan 12 16:30:58 2024 +0800
[Fix](Export) Fix the NPE exception when cancel an export job #29913
---
.../main/java/org/apache/doris/load/ExportJob.java | 27 ++
.../org/apache/doris/task/ExportExportingTask.java | 3 ++-
2 files changed, 20 insertions(+), 10 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
b/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
index 8521034f82e..0585311a1ab 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java
@@ -163,9 +163,7 @@ public class ExportJob implements Writable {
private Integer parallelNum;
-public Map getPartitionToVersion() {
-return partitionToVersion;
-}
+private Collection partitionList = new ArrayList();
private Map partitionToVersion = Maps.newHashMap();
@@ -314,7 +312,6 @@ public class ExportJob implements Writable {
List tabletIdList = Lists.newArrayList();
table.readLock();
try {
-Collection partitions = new ArrayList();
// get partitions
// user specifies partitions, already checked in ExportStmt
if (this.partitionNames != null) {
@@ -323,19 +320,18 @@ public class ExportJob implements Writable {
+ " of partitions allowed by a export job");
}
for (String partName : this.partitionNames) {
-partitions.add(table.getPartition(partName));
+partitionList.add(table.getPartition(partName));
}
} else {
if (table.getPartitions().size() >
Config.maximum_number_of_export_partitions) {
throw new UserException("The partitions number of this
export job is larger than the maximum number"
+ " of partitions allowed by a export job");
}
-partitions = table.getPartitions();
+partitionList = table.getPartitions();
}
// get tablets
-for (Partition partition : partitions) {
-partitionToVersion.put(partition.getName(),
partition.getVisibleVersion());
+for (Partition partition : partitionList) {
for (MaterializedIndex index :
partition.getMaterializedIndices(IndexExtState.VISIBLE)) {
tabletIdList.addAll(index.getTabletIdsInOrder());
}
@@ -525,6 +521,15 @@ public class ExportJob implements Writable {
this.outfileInfo = outfileInfo;
}
+public synchronized Map getPartitionToVersion() {
+if (partitionToVersion.isEmpty()) {
+// get version of partitions
+for (Partition partition : partitionList) {
+partitionToVersion.put(partition.getName(),
partition.getVisibleVersion());
+}
+}
+return partitionToVersion;
+}
public synchronized Thread getDoExportingThread() {
return doExportingThread;
@@ -582,7 +587,11 @@ public class ExportJob implements Writable {
// maybe user cancel this job
if (task != null && state == JobState.EXPORTING && stmtExecutorList !=
null) {
for (int idx = 0; idx < stmtExecutorList.size(); ++idx) {
-stmtExecutorList.get(idx).cancel();
+// because a exporting task may be cancelled due to a load
operation,
+// then it's StmtExecutor is null
+if (stmtExecutorList.get(idx) != null) {
+stmtExecutorList.get(idx).cancel();
+}
}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/task/ExportExportingTask.java
b/fe/fe-core/src/main/java/org/apache/doris/task/ExportExportingTask.java
index 142cfddd707..d5a18f19d56 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/task/ExportExportingTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/task/ExportExportingTask.java
@@ -130,6 +130,7 @@ public class ExportExportingTask extends MasterTask {
job.getTableName().getDb());
OlapTable table =
db.getOlapTableOrAnalysisException(job.getTableName().getTbl());
table.readLock();
+Map partitionToVersion =
job.getPartitionToVersion();
try {
Re: [PR] [feature](function) support ip function named ipv4_cidr_to_range(addr, cidr) [doris]
amorynan commented on code in PR #29819: URL: https://github.com/apache/doris/pull/29819#discussion_r1450039254 ## gensrc/script/doris_builtins_functions.py: ## @@ -2018,6 +2018,11 @@ [['is_ipv6_string'], 'BOOLEAN', ['STRING'], 'ALWAYS_NULLABLE'], [['is_ip_address_in_range'], 'BOOLEAN', ['VARCHAR', 'VARCHAR'], 'ALWAYS_NOT_NULLABLE'], [['is_ip_address_in_range'], 'BOOLEAN', ['STRING', 'STRING'], 'ALWAYS_NOT_NULLABLE'], +[['ipv4_cidr_to_range'], 'ANY_STRUCT_TYPE', ['IPV4', 'TINYINT'], 'ALWAYS_NULLABLE'], Review Comment: why use ANY_STRUCT_TYPE? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud rowset meta & tablet meta definition [doris]
github-actions[bot] commented on PR #29911: URL: https://github.com/apache/doris/pull/29911#issuecomment-1888653869 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](migrate disk) fix migrate disk lost data during publish version [doris]
yiguolei commented on PR #29887: URL: https://github.com/apache/doris/pull/29887#issuecomment-1888654209 run beut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipelineX](improvement) Adjust local exchange strategy [doris]
doris-robot commented on PR #29915: URL: https://github.com/apache/doris/pull/29915#issuecomment-1888654848 ClickBench: Total hot run time: 30.57 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 1c9d5a5647c5621ec9891a1742bd823cfc647327, data reload: false query1 0.050.060.06 query2 0.060.030.03 query3 0.250.110.11 query4 1.780.120.12 query5 0.520.510.57 query6 1.360.640.63 query7 0.020.010.01 query8 0.040.020.02 query9 0.550.490.50 query10 0.530.540.55 query11 0.130.090.10 query12 0.110.090.08 query13 0.610.600.59 query14 0.800.780.78 query15 0.820.810.81 query16 0.360.350.36 query17 0.960.950.92 query18 0.240.240.23 query19 1.871.741.76 query20 0.010.010.01 query21 15.42 0.570.57 query22 2.902.482.08 query23 17.09 0.710.80 query24 16.54 0.590.60 query25 2.170.140.14 query26 0.130.130.13 query27 0.140.140.14 query28 5.780.820.83 query29 12.62 3.123.14 query30 0.550.480.50 query31 2.780.350.36 query32 3.390.490.48 query33 3.233.223.21 query34 15.79 4.184.17 query35 4.184.174.19 query36 1.081.051.03 query37 0.070.050.05 query38 0.040.030.03 query39 0.020.020.02 query40 0.160.130.13 query41 0.070.010.01 query42 0.030.020.01 query43 0.030.020.01 Total cold run time: 115.28 s Total hot run time: 30.57 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [unit test](statistics)Add unit test case for auto analyze. [doris]
yiguolei merged PR #29904: URL: https://github.com/apache/doris/pull/29904 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (99ad5a83482 -> ba6d4498e3d)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 99ad5a83482 [Fix](nereids) Fix cte rewrite by mv failure and predicates compensation by mistake (#29820) add ba6d4498e3d [unit test](statistics)Add unit test case for auto analyze. #29904 No new revisions were added by this update. Summary of changes: .../doris/statistics/AnalysisManagerTest.java | 53 + .../doris/statistics/OlapAnalysisTaskTest.java | 2 +- .../suites/statistics/analyze_stats.groovy | 249 + 3 files changed, 204 insertions(+), 100 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](migrate disk) fix migrate disk lost data during publish version [doris]
yiguolei commented on PR #29887: URL: https://github.com/apache/doris/pull/29887#issuecomment-1888654346 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud rowset meta & tablet meta definition [doris]
doris-robot commented on PR #29911: URL: https://github.com/apache/doris/pull/29911#issuecomment-1888656004 ClickBench: Total hot run time: 30.29 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 0b88f15504a17501c7cf0b7e969bea5e4679fecd, data reload: false query1 0.060.060.06 query2 0.070.030.02 query3 0.260.120.12 query4 1.750.120.10 query5 0.530.520.52 query6 1.360.640.62 query7 0.010.010.01 query8 0.040.030.03 query9 0.560.490.50 query10 0.550.550.53 query11 0.120.080.09 query12 0.120.090.10 query13 0.600.590.60 query14 0.790.790.77 query15 0.820.790.80 query16 0.350.350.34 query17 0.950.990.96 query18 0.240.260.25 query19 1.791.771.68 query20 0.010.010.01 query21 15.40 0.580.57 query22 2.662.891.79 query23 17.60 0.750.77 query24 16.18 0.570.59 query25 2.170.160.14 query26 0.140.140.13 query27 0.160.150.14 query28 5.980.850.84 query29 12.61 3.173.17 query30 0.530.480.46 query31 2.770.340.36 query32 3.380.480.48 query33 3.203.223.23 query34 16.08 4.184.16 query35 4.154.194.18 query36 1.091.051.04 query37 0.070.050.06 query38 0.030.020.02 query39 0.020.020.01 query40 0.160.140.13 query41 0.080.020.01 query42 0.020.010.02 query43 0.020.020.02 Total cold run time: 115.48 s Total hot run time: 30.29 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](doriswriter)Fix the problem that specifying multiple loadurls does not take effect [doris]
yiguolei merged PR #29865: URL: https://github.com/apache/doris/pull/29865 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (ba6d4498e3d -> 05a1bbad048)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from ba6d4498e3d [unit test](statistics)Add unit test case for auto analyze. #29904 add 05a1bbad048 [fix](doriswriter)Fix the problem that specifying multiple loadurls does not take effect #29865 No new revisions were added by this update. Summary of changes: .../plugin/writer/doriswriter/DorisStreamLoadObserver.java | 10 -- 1 file changed, 4 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) adjust min/max stats for cast function if types are comparable [doris]
github-actions[bot] commented on PR #28166: URL: https://github.com/apache/doris/pull/28166#issuecomment-1888656430 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](Nereids) adjust min/max stats for cast function if types are comparable [doris]
github-actions[bot] commented on PR #28166: URL: https://github.com/apache/doris/pull/28166#issuecomment-1888656386 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [improvement](statistics)Stop analyze quickly after user close auto analyze. #29809
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new fd96e09001d [improvement](statistics)Stop analyze quickly after user
close auto analyze. #29809
fd96e09001d is described below
commit fd96e09001d56391c2f9dcd79487ec92465df922
Author: Jibing-Li <64681310+jibing...@users.noreply.github.com>
AuthorDate: Fri Jan 12 16:36:28 2024 +0800
[improvement](statistics)Stop analyze quickly after user close auto
analyze. #29809
---
.../doris/statistics/StatisticsAutoCollector.java | 26 +---
.../statistics/StatisticsAutoCollectorTest.java| 71 ++
2 files changed, 90 insertions(+), 7 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java
b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java
index 0f29d00da1a..e2eeb21aad4 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java
@@ -59,24 +59,32 @@ public class StatisticsAutoCollector extends
StatisticsCollector {
@Override
protected void collect() {
-if
(!StatisticsUtil.inAnalyzeTime(LocalTime.now(TimeUtils.getTimeZone().toZoneId(
{
-analysisTaskExecutor.clear();
-return;
-}
-if (StatisticsUtil.enableAutoAnalyze()) {
+if (canCollect()) {
analyzeAll();
}
}
-@SuppressWarnings({"rawtypes", "unchecked"})
-private void analyzeAll() {
+protected boolean canCollect() {
+return StatisticsUtil.enableAutoAnalyze()
+&&
StatisticsUtil.inAnalyzeTime(LocalTime.now(TimeUtils.getTimeZone().toZoneId()));
+}
+
+protected void analyzeAll() {
List catalogs = getCatalogsInOrder();
for (CatalogIf ctl : catalogs) {
+if (!canCollect()) {
+analysisTaskExecutor.clear();
+break;
+}
if (!ctl.enableAutoAnalyze()) {
continue;
}
List dbs = getDatabasesInOrder(ctl);
for (DatabaseIf databaseIf : dbs) {
+if (!canCollect()) {
+analysisTaskExecutor.clear();
+break;
+}
if
(StatisticConstants.SYSTEM_DBS.contains(databaseIf.getFullName())) {
continue;
}
@@ -109,6 +117,10 @@ public class StatisticsAutoCollector extends
StatisticsCollector {
List analysisInfos = constructAnalysisInfo(databaseIf);
for (AnalysisInfo analysisInfo : analysisInfos) {
try {
+if (!canCollect()) {
+analysisTaskExecutor.clear();
+break;
+}
analysisInfo = getReAnalyzeRequiredPart(analysisInfo);
if (analysisInfo == null) {
continue;
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/statistics/StatisticsAutoCollectorTest.java
b/fe/fe-core/src/test/java/org/apache/doris/statistics/StatisticsAutoCollectorTest.java
index 68f83fa3e30..2455581c29c 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/statistics/StatisticsAutoCollectorTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/statistics/StatisticsAutoCollectorTest.java
@@ -31,6 +31,7 @@ import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.FeConstants;
import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.statistics.AnalysisInfo.AnalysisMethod;
import org.apache.doris.statistics.AnalysisInfo.AnalysisType;
import org.apache.doris.statistics.AnalysisInfo.JobType;
@@ -460,4 +461,74 @@ public class StatisticsAutoCollectorTest {
Assertions.assertNotNull(task.getTableSample());
}
}
+
+@Test
+public void testDisableAuto1() throws Exception {
+InternalCatalog catalog1 = new InternalCatalog();
+List catalogs = Lists.newArrayList();
+catalogs.add(catalog1);
+
+new MockUp() {
+@Mock
+public List getCatalogsInOrder() {
+return catalogs;
+}
+
+@Mock
+protected boolean canCollect() {
+return false;
+}
+
+};
+
+StatisticsAutoCollector sac = new StatisticsAutoCollector();
+new Expectations(catalog1) {{
+catalog1.enableAutoAnalyze();
+times = 0;
+}};
+
+sac.analyzeAll();
+}
+
+@Test
+public void testDisableAuto2() throws Exception {
+InternalCatalog catalog1 = new InternalCatalog();
Re: [PR] [improvement](statistics)Stop analyze quickly after user close auto analyze. [doris]
yiguolei merged PR #29809: URL: https://github.com/apache/doris/pull/29809 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [typo](docs) fix typo of outfile and export md [doris]
yiguolei merged PR #29804: URL: https://github.com/apache/doris/pull/29804 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (fd96e09001d -> 2e6c5b9a4d0)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from fd96e09001d [improvement](statistics)Stop analyze quickly after user close auto analyze. #29809 add 2e6c5b9a4d0 [typo](docs) fix typo of outfile and export md (#29804) No new revisions were added by this update. Summary of changes: .../sql-reference/Data-Manipulation-Statements/Manipulation/EXPORT.md | 2 +- .../sql-manual/sql-reference/Data-Manipulation-Statements/OUTFILE.md | 4 ++-- .../sql-reference/Data-Manipulation-Statements/Manipulation/EXPORT.md | 2 +- .../sql-manual/sql-reference/Data-Manipulation-Statements/OUTFILE.md | 4 ++-- 4 files changed, 6 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](auth)Not allowed to operate internal_schema database [doris]
yiguolei merged PR #29790: URL: https://github.com/apache/doris/pull/29790 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improvement](aggregate) test [doris]
github-actions[bot] commented on PR #29919: URL: https://github.com/apache/doris/pull/29919#issuecomment-1888658072 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (2e6c5b9a4d0 -> f929de3a1e0)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from 2e6c5b9a4d0 [typo](docs) fix typo of outfile and export md (#29804)
add f929de3a1e0 [improvement](auth)Not allowed to operate internal_schema
database (#29790)
No new revisions were added by this update.
Summary of changes:
.../doris/analysis/AlterDatabasePropertyStmt.java | 3 +-
.../doris/analysis/AlterDatabaseQuotaStmt.java | 3 +-
.../apache/doris/analysis/AlterDatabaseRename.java | 3 +-
.../org/apache/doris/analysis/AlterTableStmt.java | 2 +
.../org/apache/doris/analysis/CreateDbStmt.java| 3 +-
.../org/apache/doris/analysis/CreateTableStmt.java | 3 +-
.../java/org/apache/doris/analysis/DropDbStmt.java | 3 +-
.../org/apache/doris/analysis/DropTableStmt.java | 3 +-
.../doris/analysis/InsertOverwriteTableStmt.java | 3 +
.../apache/doris/analysis/TruncateTableStmt.java | 3 +-
.../doris/common/util/InternalDatabaseUtil.java} | 33 ++
.../trees/plans/commands/DeleteFromCommand.java| 10 ++
.../commands/InsertOverwriteTableCommand.java | 3 +
.../trees/plans/commands/info/CreateTableInfo.java | 7 +-
.../doris/transaction/DatabaseTransactionMgr.java | 3 +
.../operate_internal_schema.groovy | 126 +
16 files changed, 180 insertions(+), 31 deletions(-)
copy
fe/fe-core/src/{test/java/org/apache/doris/analysis/IsNullPredicateTest.java =>
main/java/org/apache/doris/common/util/InternalDatabaseUtil.java} (56%)
create mode 100644
regression-test/suites/internal_schema_p0/operate_internal_schema.groovy
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](dialect) make http sql converter plugin and audit loader as builtin plugin [doris]
morningman commented on code in PR #29692:
URL: https://github.com/apache/doris/pull/29692#discussion_r1450045212
##
fe/fe-core/src/main/java/org/apache/doris/catalog/InternalSchemaInitializer.java:
##
@@ -83,6 +113,7 @@ public void run() {
Database database = op.get();
modifyTblReplicaCount(database, StatisticConstants.STATISTIC_TBL_NAME);
modifyTblReplicaCount(database, StatisticConstants.HISTOGRAM_TBL_NAME);
+modifyTblReplicaCount(database, AuditLoaderPlugin.AUDIT_LOG_TABLE);
Review Comment:
I will try to refactor this later, there are some other things need to be
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipelineX](improvement) Adjust local exchange strategy [doris]
doris-robot commented on PR #29915: URL: https://github.com/apache/doris/pull/29915#issuecomment-1888659021 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 1c9d5a5647c5621ec9891a1742bd823cfc647327 with default session variables Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s Insert into select: 12.2 seconds inserted 1000 Rows, about 819K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](dialect) make http sql converter plugin and audit loader as builtin plugin [doris]
morningman commented on code in PR #29692:
URL: https://github.com/apache/doris/pull/29692#discussion_r1450044610
##
fe/fe-core/src/main/java/org/apache/doris/httpv2/rest/LoadAction.java:
##
@@ -439,8 +433,10 @@ private Object executeWithClusterToken(HttpServletRequest
request, String db,
return redirectView;
} catch (Exception e) {
-LOG.warn("Failed to execute stream load with cluster token, {}",
e);
+LOG.warn("Failed to execute stream load with cluster token, {}",
e.getMessage(), e);
Review Comment:
my way is more friendly for `grep`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud rowset meta & tablet meta definition [doris]
github-actions[bot] commented on PR #29911: URL: https://github.com/apache/doris/pull/29911#issuecomment-1888658815 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [log](insert-into) add log in parse insert into values data [doris]
github-actions[bot] commented on PR #29903: URL: https://github.com/apache/doris/pull/29903#issuecomment-1888660383 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [log](insert-into) add log in parse insert into values data [doris]
github-actions[bot] commented on PR #29903: URL: https://github.com/apache/doris/pull/29903#issuecomment-1888660466 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](migrate disk) fix tablet disk migration timeout too large [doris]
github-actions[bot] commented on PR #29895: URL: https://github.com/apache/doris/pull/29895#issuecomment-1888660785 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](merge-cloud) Add cloud rowset meta & tablet meta definition [doris]
doris-robot commented on PR #29911: URL: https://github.com/apache/doris/pull/29911#issuecomment-1888660904 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 0b88f15504a17501c7cf0b7e969bea5e4679fecd with default session variables Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s Insert into select: 13.5 seconds inserted 1000 Rows, about 740K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](http) fix exception when querying map data through http [doris]
yiguolei merged PR #29686: URL: https://github.com/apache/doris/pull/29686 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (f929de3a1e0 -> a446f8425c0)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from f929de3a1e0 [improvement](auth)Not allowed to operate internal_schema database (#29790) add a446f8425c0 [fix](http) fix exception when querying map data through http #29686 No new revisions were added by this update. Summary of changes: fe/fe-common/src/main/java/org/apache/doris/catalog/PrimitiveType.java | 2 -- 1 file changed, 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](migrate disk) fix tablet disk migration timeout too large [doris]
github-actions[bot] commented on PR #29895: URL: https://github.com/apache/doris/pull/29895#issuecomment-1888660851 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [deps](hadoop) upgrade hadoop deps to 3.3.4.6 [doris]
github-actions[bot] commented on PR #29908: URL: https://github.com/apache/doris/pull/29908#issuecomment-1888662880 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](multi-table-load) pause job when can not find table [doris]
github-actions[bot] commented on PR #29870: URL: https://github.com/apache/doris/pull/29870#issuecomment-1888662797 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature][cdc] specify bucket when synchronizing database [doris-flink-connector]
JNSimba commented on code in PR #288:
URL:
https://github.com/apache/doris-flink-connector/pull/288#discussion_r1450050429
##
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java:
##
@@ -338,6 +350,60 @@ protected HashMap multiToOneRulesParser(
return multiToOneRulesPattern;
}
+/**
+ * Get table buckets Map.
+ *
+ * @param tableBuckets the string of tableBuckets,
eg:student:10,student_info:20,student.*:30
+ * @return The table name and buckets map. The key is table name, the
value is buckets.
+ */
+public Map getTableBuckets(String tableBuckets) {
+Map tableBucketsMap = new LinkedHashMap<>();
+String[] tableBucketsArray = tableBuckets.split(",");
+for (String tableBucket : tableBucketsArray) {
+String[] tableBucketArray = tableBucket.split(":");
+tableBucketsMap.put(
+tableBucketArray[0].trim(),
Integer.parseInt(tableBucketArray[1].trim()));
+}
+return tableBucketsMap;
+}
+
+/**
+ * Set table schema buckets.
+ *
+ * @param tableBucketsMap The table name and buckets map. The key is table
name, the value is
+ * buckets.
+ * @param dorisSchema @{TableSchema}
+ * @param dorisTable the table name need to set buckets
+ * @param tableHasSet The buckets table is set
+ */
+public void setTableSchemaBuckets(
+Map tableBucketsMap,
+TableSchema dorisSchema,
+String dorisTable,
+Set tableHasSet) {
+
+if (tableBucketsMap != null) {
+// Firstly, if the table name is in the table-buckets map, set the
buckets of the table.
+if (tableBucketsMap.containsKey(dorisTable)) {
+dorisSchema.setTableBuckets(tableBucketsMap.get(dorisTable));
+tableHasSet.add(dorisTable);
+return;
+}
+// Secondly, iterate over the map to find a corresponding regular
expression match,
+for (Map.Entry entry :
tableBucketsMap.entrySet()) {
+if (tableHasSet.contains(entry.getKey())) {
+continue;
+}
+
+Pattern pattern = Pattern.compile(entry.getKey());
+if (pattern.matcher(dorisTable).matches()) {
+dorisSchema.setTableBuckets(entry.getValue());
+return;
Review Comment:
Isn’t it necessary to put tableHasSet here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](serde)update slice reserve and deduce slice back usage [doris]
doris-robot commented on PR #29879: URL: https://github.com/apache/doris/pull/29879#issuecomment-1888664719 TPC-DS: Total hot run time: 178760 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit aa04f7a1e74623d30588fcad3ccaf8a149af9190, data reload: false query1 940 335 328 328 query2 6714189018901890 query3 6708211 205 205 query4 26085 22242 22255 22242 query5 7200592 548 548 query6 258 187 191 187 query7 4615277 270 270 query8 227 201 200 200 query9 9282274028112740 query10 594 243 235 235 query11 16156 15444 15217 15217 query12 125 71 72 71 query13 1691389 391 389 query14 11870 726272117211 query15 261 189 196 189 query16 5376255 239 239 query17 1921484 479 479 query18 1952277 260 260 query19 285 150 150 150 query20 75 81 72 72 query21 188 131 122 122 query22 5291501249454945 query23 31741 30964 30955 30955 query24 12327 283428152815 query25 546 332 329 329 query26 1803150 157 150 query27 2971278 284 278 query28 7135186818451845 query29 2121687 689 687 query30 287 136 141 136 query31 970 761 771 761 query32 89 64 54 54 query33 714 242 246 242 query34 1148466 472 466 query35 898 735 739 735 query36 1341120712231207 query37 188 68 67 67 query38 3431328632773277 query39 1345128312741274 query40 336 101 92 92 query41 38 35 35 35 query42 102 96 98 96 query43 515 499 511 499 query44 1070703 706 703 query45 208 189 187 187 query46 1063649 656 649 query47 1639149916041499 query48 418 336 334 334 query49 1283316 294 294 query50 707 331 324 324 query51 5282523452585234 query52 102 86 95 86 query53 359 289 282 282 query54 901 450 464 450 query55 93 83 88 83 query56 208 195 193 193 query57 985 973 955 955 query58 220 200 194 194 query59 2741248225492482 query60 224 213 217 213 query61 82 87 80 80 query62 672 396 382 382 query63 307 283 279 279 query64 5774166716711667 query65 3352325932793259 query66 1410334 341 334 query67 15841 15381 15268 15268 query68 12325 515 509 509 query69 674 394 379 379 query70 1724151214761476 query71 545 237 251 237 query72 4867282828182818 query73 2642312 313 312 query74 6966644865106448 query75 5190227623122276 query76 6297908 1093908 query77 677 264 292 264 query78 9831878187328732 query79 1010525 501 501 query80 558 354 343 343 query81 456 205 206 205 query82 214 92 94 92 query83 142 116 121 116 query84 280 78 75 75 query85 1047333 322 322 query86 372 373 384 373 query87 3591339633753375 query88 2989218621922186 query89 444 372 378 372 query90 2062209 194 194 query91 168 121 125 121 query92 63 55 51 51 query93 2062445 427 427 query94 1181176 177 176 query95 570 472 475 472 query96 622 320 306 306 query97 4263417842134178 query98 205 195 191 191 query99 978 705 734 705 Total cold run time: 297816 ms Total hot run time: 178760 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the s
Re: [PR] [enhance](serde)update slice reserve and deduce slice back usage [doris]
doris-robot commented on PR #29879: URL: https://github.com/apache/doris/pull/29879#issuecomment-1888666533 (From new machine)TeamCity pipeline, clickbench performance test result: the sum of best hot time: 46.31 seconds stream load tsv: 560 seconds loaded 74807831229 Bytes, about 127 MB/s stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s insert into select: 21.4 seconds inserted 1000 Rows, about 467K ops/s storage size: 17183831004 Bytes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature][cdc] specify bucket when synchronizing database [doris-flink-connector]
JNSimba commented on code in PR #288:
URL:
https://github.com/apache/doris-flink-connector/pull/288#discussion_r1450053836
##
flink-doris-connector/src/main/java/org/apache/doris/flink/tools/cdc/DatabaseSync.java:
##
@@ -338,6 +350,60 @@ protected HashMap multiToOneRulesParser(
return multiToOneRulesPattern;
}
+/**
+ * Get table buckets Map.
+ *
+ * @param tableBuckets the string of tableBuckets,
eg:student:10,student_info:20,student.*:30
+ * @return The table name and buckets map. The key is table name, the
value is buckets.
+ */
+public Map getTableBuckets(String tableBuckets) {
+Map tableBucketsMap = new LinkedHashMap<>();
+String[] tableBucketsArray = tableBuckets.split(",");
+for (String tableBucket : tableBucketsArray) {
+String[] tableBucketArray = tableBucket.split(":");
+tableBucketsMap.put(
+tableBucketArray[0].trim(),
Integer.parseInt(tableBucketArray[1].trim()));
+}
+return tableBucketsMap;
+}
+
+/**
+ * Set table schema buckets.
+ *
+ * @param tableBucketsMap The table name and buckets map. The key is table
name, the value is
+ * buckets.
+ * @param dorisSchema @{TableSchema}
+ * @param dorisTable the table name need to set buckets
+ * @param tableHasSet The buckets table is set
+ */
+public void setTableSchemaBuckets(
+Map tableBucketsMap,
+TableSchema dorisSchema,
+String dorisTable,
+Set tableHasSet) {
+
+if (tableBucketsMap != null) {
+// Firstly, if the table name is in the table-buckets map, set the
buckets of the table.
+if (tableBucketsMap.containsKey(dorisTable)) {
+dorisSchema.setTableBuckets(tableBucketsMap.get(dorisTable));
+tableHasSet.add(dorisTable);
+return;
+}
+// Secondly, iterate over the map to find a corresponding regular
expression match,
+for (Map.Entry entry :
tableBucketsMap.entrySet()) {
+if (tableHasSet.contains(entry.getKey())) {
+continue;
+}
+
+Pattern pattern = Pattern.compile(entry.getKey());
+if (pattern.matcher(dorisTable).matches()) {
Review Comment:
Do we need to consider tablePrefix and tableSuffix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [log](insert-into) add log in parse insert into values data [doris]
HHoflittlefish777 commented on PR #29903: URL: https://github.com/apache/doris/pull/29903#issuecomment-1888672036 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve](multi-table-load) pause job when can not find table [doris]
yiguolei merged PR #29870: URL: https://github.com/apache/doris/pull/29870 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (ce045c793d8 -> 4590c841fb5)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from ce045c793d8 [improve](multi-table-load) pause job when can not find table #29870 add 4590c841fb5 [feat](meta) Reuse HMS statistics analyzed by Spark engine for Analyze Task. (#28525) No new revisions were added by this update. Summary of changes: .../doris/catalog/external/HMSExternalTable.java | 49 +- .../apache/doris/statistics/HMSAnalysisTask.java | 41 -- 2 files changed, 85 insertions(+), 5 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (a446f8425c0 -> ce045c793d8)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from a446f8425c0 [fix](http) fix exception when querying map data through http #29686 add ce045c793d8 [improve](multi-table-load) pause job when can not find table #29870 No new revisions were added by this update. Summary of changes: .../apache/doris/service/FrontendServiceImpl.java | 11 +++ .../data/multi_table_load_invalid_table.csv| 1 + .../routine_load/test_routine_load_error.groovy| 94 ++ 3 files changed, 106 insertions(+) create mode 100644 regression-test/suites/load_p0/routine_load/data/multi_table_load_invalid_table.csv create mode 100644 regression-test/suites/load_p0/routine_load/test_routine_load_error.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](serde)update slice reserve and deduce slice back usage [doris]
doris-robot commented on PR #29879: URL: https://github.com/apache/doris/pull/29879#issuecomment-1888673382 ClickBench: Total hot run time: 30.53 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit aa04f7a1e74623d30588fcad3ccaf8a149af9190, data reload: false query1 0.060.050.05 query2 0.060.030.02 query3 0.250.110.11 query4 1.790.120.12 query5 0.530.520.52 query6 1.340.640.63 query7 0.010.010.01 query8 0.040.030.02 query9 0.550.490.49 query10 0.560.550.54 query11 0.120.090.10 query12 0.120.090.09 query13 0.610.600.60 query14 0.790.770.78 query15 0.810.800.81 query16 0.340.340.36 query17 1.001.010.97 query18 0.240.240.24 query19 1.821.731.74 query20 0.010.010.01 query21 15.41 0.550.57 query22 2.412.531.88 query23 17.19 0.790.78 query24 16.21 0.590.57 query25 2.170.150.14 query26 0.130.140.13 query27 0.140.140.15 query28 6.030.810.80 query29 12.58 3.343.30 query30 0.520.470.48 query31 2.780.350.36 query32 3.400.490.49 query33 3.223.213.21 query34 15.84 4.134.11 query35 4.164.204.16 query36 1.081.051.05 query37 0.060.050.05 query38 0.030.030.03 query39 0.020.010.02 query40 0.150.120.14 query41 0.070.020.01 query42 0.020.020.01 query43 0.020.020.02 Total cold run time: 114.69 s Total hot run time: 30.53 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feat](stats) Reuse HMS statistics analyzed by Spark engine for Analyze [doris]
yiguolei merged PR #28525: URL: https://github.com/apache/doris/pull/28525 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](create tablet) backend create tablet round robin among … [doris]
dataroaring commented on code in PR #29818:
URL: https://github.com/apache/doris/pull/29818#discussion_r1450059347
##
be/src/olap/data_dir.cpp:
##
@@ -891,6 +891,10 @@ void DataDir::update_remote_data_size(int64_t size) {
disks_remote_used_capacity->set_value(size);
}
+size_t DataDir::disk_available() const {
+return _available_bytes;
+}
Review Comment:
available_bytes() is a better name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [doc](unique key) update description for unique key in data model doc [doris]
yiguolei merged PR #28902: URL: https://github.com/apache/doris/pull/28902 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (4590c841fb5 -> c87f4523ccd)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from 4590c841fb5 [feat](meta) Reuse HMS statistics analyzed by Spark engine for Analyze Task. (#28525) add c87f4523ccd [doc](unique key) update description for unique key in data model doc (#28902) No new revisions were added by this update. Summary of changes: docs/en/docs/data-table/data-model.md| 37 docs/zh-CN/docs/data-table/data-model.md | 32 ++- 2 files changed, 45 insertions(+), 24 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Error while running github feature from .asf.yaml in doris!
An error occurred while running github feature in .asf.yaml!: You can only have a maximum of 10 external triage collaborators, please contact vp-in...@apache.org to request an exception. - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](serde)update slice reserve and deduce slice back usage [doris]
doris-robot commented on PR #29879: URL: https://github.com/apache/doris/pull/29879#issuecomment-1888677784 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit aa04f7a1e74623d30588fcad3ccaf8a149af9190 with default session variables Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s Insert into select: 13.1 seconds inserted 1000 Rows, about 763K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipelineX](improvement) Adjust local exchange strategy [doris]
github-actions[bot] commented on PR #29915: URL: https://github.com/apache/doris/pull/29915#issuecomment-1888677821 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](dialect) make http sql converter plugin and audit loader as builtin plugin [doris]
morningman commented on PR #29692: URL: https://github.com/apache/doris/pull/29692#issuecomment-1888678107 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](dialect) make http sql converter plugin and audit loader as builtin plugin [doris]
morningman commented on code in PR #29692:
URL: https://github.com/apache/doris/pull/29692#discussion_r1450063105
##
fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java:
##
@@ -104,7 +110,22 @@ public final class GlobalVariable {
@VariableMgr.VarAttr(name = SHOW_FULL_DBNAME_IN_INFO_SCHEMA_DB, flag =
VariableMgr.GLOBAL)
public static boolean showFullDbNameInInfoSchemaDb = false;
-// Don't allow to create instance.
+@VariableMgr.VarAttr(name = SQL_CONVERTER_SERVICE, flag =
VariableMgr.GLOBAL)
+public static String sqlConverterService = "";
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](function) support ip function named ipv4_cidr_to_range(addr, cidr) [doris]
amorynan commented on code in PR #29819:
URL: https://github.com/apache/doris/pull/29819#discussion_r1450072812
##
be/src/vec/functions/function_ip.h:
##
@@ -832,4 +845,170 @@ class FunctionIsIPAddressInRange : public IFunction {
}
};
+class FunctionIPv4CIDRToRange : public IFunction {
+private:
+static inline std::pair apply_cidr_mask(UInt32 src, UInt8
bits_to_keep) {
+if (bits_to_keep >= 8 * sizeof(UInt32)) {
+return {src, src};
+}
+if (bits_to_keep == 0) {
+return {static_cast(0), static_cast(-1)};
+}
+UInt32 mask = static_cast(-1) << (8 * sizeof(UInt32) -
bits_to_keep);
+UInt32 lower = src & mask;
+UInt32 upper = lower | ~mask;
+
+return {lower, upper};
+}
+
+template
+Status execute_type(Block& block, const ColumnsWithTypeAndName& argments,
size_t result) const {
+auto ip_argment = argments[0];
+auto cidr_argment = argments[1];
+using ColumnType = ColumnVector;
+const ColumnPtr& ip_column = ip_argment.column;
+const ColumnPtr& cidr_column = cidr_argment.column;
+DCHECK(ip_column->size() == cidr_column->size());
+size_t col_size = ip_column->size();
+
+ColumnPtr argument_nullmap[2] = {nullptr, nullptr};
+
+if (ip_column->is_nullable()) {
+const auto* col_ip_nullable =
check_and_get_column(ip_column.get());
+if (!col_ip_nullable) {
+return Status::InvalidArgument("Illegal column {} of first
argument of function {}",
+ ip_column->get_name(),
get_name());
+}
+argument_nullmap[0] = col_ip_nullable->get_null_map_column_ptr();
+}
+
+if (cidr_column->is_nullable()) {
+const auto* col_cidr_nullable =
check_and_get_column(ip_column.get());
+if (!col_cidr_nullable) {
+return Status::InvalidArgument(
+"Illegal column {} of second argument of function {}",
+ip_column->get_name(), get_name());
+}
+argument_nullmap[1] = col_cidr_nullable->get_null_map_column_ptr();
+}
+
+const auto* col_ip_column =
check_and_get_column(ip_column.get());
+if (!col_ip_column) {
+return Status::InvalidArgument("Illegal column {} of first
argument of function {}",
+ ip_column->get_name(), get_name());
+}
+
+const auto* col_cidr_column =
check_and_get_column>(cidr_column.get());
Review Comment:
use data type to check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
