Re: [PR] [improvement](decommission be) decommission check replica num #32748 [doris]
doris-robot commented on PR #34038: URL: https://github.com/apache/doris/pull/34038#issuecomment-2099878559 ClickBench: Total hot run time: 30.97 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit fddf6456d2dbe43d18635802dee024a051d5a46b, data reload: false query1 0.020.030.03 query2 0.070.020.02 query3 0.240.050.05 query4 1.810.060.07 query5 0.530.530.52 query6 1.250.630.62 query7 0.020.010.02 query8 0.030.020.02 query9 0.530.480.49 query10 0.520.530.55 query11 0.120.090.08 query12 0.120.090.09 query13 0.630.620.62 query14 0.780.790.79 query15 0.780.760.75 query16 0.360.380.37 query17 1.021.020.98 query18 0.220.260.25 query19 1.941.761.82 query20 0.020.020.02 query21 15.45 0.540.55 query22 2.092.531.92 query23 17.36 1.120.79 query24 4.871.521.19 query25 0.370.060.05 query26 0.660.170.15 query27 0.050.040.04 query28 7.580.740.77 query29 12.78 2.372.20 query30 0.660.530.50 query31 2.810.390.38 query32 3.360.490.49 query33 3.093.153.05 query34 15.27 4.794.82 query35 4.904.834.84 query36 1.051.021.01 query37 0.060.050.04 query38 0.040.020.02 query39 0.020.010.02 query40 0.160.140.15 query41 0.070.010.01 query42 0.020.020.02 query43 0.020.010.01 Total cold run time: 103.75 s Total hot run time: 30.97 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Planner) add date/month/year add function fe constant fold with dateV2/dateTimeV2 [doris]
doris-robot commented on PR #34484: URL: https://github.com/apache/doris/pull/34484#issuecomment-2099878582 ClickBench: Total hot run time: 30.71 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 20e9e6fdb185a71294229d7d3293d32f71a30756, data reload: false query1 0.020.020.02 query2 0.060.020.02 query3 0.250.050.05 query4 1.790.090.07 query5 0.540.520.53 query6 1.240.620.62 query7 0.020.010.01 query8 0.020.020.02 query9 0.530.470.48 query10 0.540.540.53 query11 0.120.090.08 query12 0.110.090.09 query13 0.610.610.62 query14 0.790.770.80 query15 0.800.750.78 query16 0.370.370.36 query17 1.041.031.00 query18 0.210.280.23 query19 1.921.791.87 query20 0.020.010.01 query21 15.48 0.550.52 query22 2.222.372.04 query23 17.19 0.950.83 query24 8.440.850.73 query25 0.410.110.05 query26 0.860.160.16 query27 0.050.030.04 query28 5.190.760.79 query29 12.63 2.352.21 query30 0.550.530.53 query31 2.820.400.37 query32 3.390.500.50 query33 3.073.083.09 query34 15.28 4.814.83 query35 4.874.814.85 query36 1.051.021.02 query37 0.060.040.05 query38 0.030.020.02 query39 0.020.010.02 query40 0.160.140.14 query41 0.070.020.01 query42 0.020.010.01 query43 0.020.020.01 Total cold run time: 104.88 s Total hot run time: 30.71 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](merge-on-write) remove some CHECKs in Tablet::revise_tablet_meta [doris]
dataroaring merged PR #31268: URL: https://github.com/apache/doris/pull/31268 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](merge-on-write) remove some CHECKs in Tablet::revise_tablet_meta (#31268)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f1b6bc4da99 [fix](merge-on-write) remove some CHECKs in
Tablet::revise_tablet_meta (#31268)
f1b6bc4da99 is described below
commit f1b6bc4da9917c38109df8c87e3ad5e773dedaff
Author: zhannngchen <48427519+zhannngc...@users.noreply.github.com>
AuthorDate: Wed May 8 15:02:41 2024 +0800
[fix](merge-on-write) remove some CHECKs in Tablet::revise_tablet_meta
(#31268)
---
be/src/olap/base_tablet.cpp | 31 -
be/src/olap/base_tablet.h | 5 +--
be/src/olap/tablet.cpp | 85 ++---
3 files changed, 99 insertions(+), 22 deletions(-)
diff --git a/be/src/olap/base_tablet.cpp b/be/src/olap/base_tablet.cpp
index 74c8810e26e..aec5495262d 100644
--- a/be/src/olap/base_tablet.cpp
+++ b/be/src/olap/base_tablet.cpp
@@ -1368,13 +1368,19 @@ Status BaseTablet::check_rowid_conversion(
}
// The caller should hold _rowset_update_lock and _meta_lock lock.
-Status BaseTablet::update_delete_bitmap_without_lock(const BaseTabletSPtr&
self,
- const RowsetSharedPtr&
rowset) {
-DBUG_EXECUTE_IF("Tablet.update_delete_bitmap_without_lock.random_failed", {
-if (rand() % 100 < (100 * dp->param("percent", 0.1))) {
-
LOG_WARNING("Tablet.update_delete_bitmap_without_lock.random_failed");
+Status BaseTablet::update_delete_bitmap_without_lock(
+const BaseTabletSPtr& self, const RowsetSharedPtr& rowset,
+const std::vector* specified_base_rowsets) {
+
DBUG_EXECUTE_IF("BaseTablet.update_delete_bitmap_without_lock.random_failed", {
+auto rnd = rand() % 100;
+auto percent = dp->param("percent", 0.1);
+if (rnd < (100 * percent)) {
+LOG(WARNING) <<
"BaseTablet.update_delete_bitmap_without_lock.random_failed";
return Status::InternalError(
"debug tablet update delete bitmap without lock random
failed");
+} else {
+LOG(INFO) <<
"BaseTablet.update_delete_bitmap_without_lock.random_failed not triggered"
+ << ", rnd:" << rnd << ", percent: " << percent;
}
});
int64_t cur_version = rowset->end_version();
@@ -1387,12 +1393,21 @@ Status
BaseTablet::update_delete_bitmap_without_lock(const BaseTabletSPtr& self,
<< self->tablet_id() << " cur max_version: " << cur_version;
return Status::OK();
}
-RowsetIdUnorderedSet cur_rowset_ids;
-RETURN_IF_ERROR(self->get_all_rs_id_unlocked(cur_version - 1,
&cur_rowset_ids));
+
+// calculate delete bitmap between segments if necessary.
DeleteBitmapPtr delete_bitmap =
std::make_shared(self->tablet_id());
RETURN_IF_ERROR(self->calc_delete_bitmap_between_segments(rowset,
segments, delete_bitmap));
-std::vector specified_rowsets =
self->get_rowset_by_ids(&cur_rowset_ids);
+// get all base rowsets to calculate on
+std::vector specified_rowsets;
+RowsetIdUnorderedSet cur_rowset_ids;
+if (specified_base_rowsets == nullptr) {
+RETURN_IF_ERROR(self->get_all_rs_id_unlocked(cur_version - 1,
&cur_rowset_ids));
+specified_rowsets = self->get_rowset_by_ids(&cur_rowset_ids);
+} else {
+specified_rowsets = *specified_base_rowsets;
+}
+
OlapStopWatch watch;
auto token = self->calc_delete_bitmap_executor()->create_token();
RETURN_IF_ERROR(calc_delete_bitmap(self, rowset, segments,
specified_rowsets, delete_bitmap,
diff --git a/be/src/olap/base_tablet.h b/be/src/olap/base_tablet.h
index ea36ca8c47a..a157cd1e018 100644
--- a/be/src/olap/base_tablet.h
+++ b/be/src/olap/base_tablet.h
@@ -230,8 +230,9 @@ public:
const std::map>>&
location_map);
-static Status update_delete_bitmap_without_lock(const BaseTabletSPtr& self,
-const RowsetSharedPtr&
rowset);
+static Status update_delete_bitmap_without_lock(
+const BaseTabletSPtr& self, const RowsetSharedPtr& rowset,
+const std::vector* specified_base_rowsets =
nullptr);
// end MoW functions
diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp
index dfe1220ada8..a6f0202d1e3 100644
--- a/be/src/olap/tablet.cpp
+++ b/be/src/olap/tablet.cpp
@@ -330,11 +330,19 @@ Status Tablet::revise_tablet_meta(const
std::vector& to_add,
const std::vector&
to_delete,
bool is_incremental_clone) {
LOG(INFO) << "begin to revise tablet. tablet_id=" << tablet_id();
-delete_rowsets(to_delete, false);
-add_rowsets(to_add);
-// reconstruct from tablet meta
-
Re: [PR] [improvement](decommission be) decommission check replica num #32748 [doris]
doris-robot commented on PR #34038: URL: https://github.com/apache/doris/pull/34038#issuecomment-2099883248 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit fddf6456d2dbe43d18635802dee024a051d5a46b with default session variables Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s Insert into select: 21.4 seconds inserted 1000 Rows, about 467K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](Planner) add date/month/year add function fe constant fold with dateV2/dateTimeV2 [doris]
doris-robot commented on PR #34484: URL: https://github.com/apache/doris/pull/34484#issuecomment-2099883375 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit 20e9e6fdb185a71294229d7d3293d32f71a30756 with default session variables Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s Insert into select: 21.4 seconds inserted 1000 Rows, about 467K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Enhencement](trino-connector) The trino-connector catalog supports pushdown predicates to the connector [doris]
BePPPower commented on PR #34422: URL: https://github.com/apache/doris/pull/34422#issuecomment-2099885043 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Fix](trino-connector) refactor code of TrinoConnectorExternalTable [doris]
BePPPower commented on PR #34496: URL: https://github.com/apache/doris/pull/34496#issuecomment-2099885442 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](Nereids) support where, group by, having, order by clause without from clause in query statement [doris]
starocean999 commented on PR #34475: URL: https://github.com/apache/doris/pull/34475#issuecomment-2099885470 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] test [doris]
github-actions[bot] commented on PR #34394: URL: https://github.com/apache/doris/pull/34394#issuecomment-2099887970 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] test [doris]
zzzxl1993 commented on PR #34394: URL: https://github.com/apache/doris/pull/34394#issuecomment-2099889271 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](file-scanner) print current path when encountering error (#34365) [doris]
doris-robot commented on PR #34524: URL: https://github.com/apache/doris/pull/34524#issuecomment-2099894868 TPC-H: Total hot run time: 49651 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit dc88f0a0a35e74ad45a474eef1bbc12bee676638, data reload: false -- Round 1 -- q1 17746 441243574357 q2 2027153 141 141 q3 10466 186519171865 q4 10359 123213151232 q5 8640392339153915 q6 226 123 121 121 q7 2034160315981598 q8 9297271327132713 q9 10654 10355 10303 10303 q10 8610355134753475 q11 419 242 247 242 q12 470 307 306 306 q13 18348 392739893927 q14 346 319 325 319 q15 498 455 457 455 q16 695 581 568 568 q17 1125949 963 949 q18 7207701668586858 q19 1710154015441540 q20 512 317 305 305 q21 4489407341134073 q22 487 402 389 389 Total cold run time: 116365 ms Total hot run time: 49651 ms - Round 2, with runtime_filter_mode=off - q1 4283427042894270 q2 319 219 220 219 q3 4129414941594149 q4 2736273827662738 q5 7200716671187118 q6 230 119 119 119 q7 3214287028222822 q8 4340445244564452 q9 16818 16697 16764 16697 q10 4219426142554255 q11 740 677 691 677 q12 1046849 862 849 q13 7291374237273727 q14 435 440 411 411 q15 501 460 463 460 q16 727 681 679 679 q17 3825382838333828 q18 8891871388208713 q19 1718169516511651 q20 2398212621122112 q21 8522856284528452 q22 1012986 922 922 Total cold run time: 84594 ms Total hot run time: 79320 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix] Fix the bug that the `DorisTypeMapper` datetime type will lose precision when converted to flink type [doris-flink-connector]
huyuanfeng2018 commented on PR #379: URL: https://github.com/apache/doris-flink-connector/pull/379#issuecomment-2099895708 > How was that triggered? In theory, the catalog will reach this point Then the doris table is created through the doris command and then written through flink  As shown in the figure, the stime field is at the millisecond level. I write it through flink. ``` USE CATALOG doris; insert into doris.db,table xxx ``` The `DorisCatalog#getTable` method will return the mapping of the doris table to the flink table. The precision is not considered when processing the datetime type.  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] fix RLE count larger than int32 unit test [doris]
133tosakarin closed pull request #34525: fix RLE count larger than int32 unit test URL: https://github.com/apache/doris/pull/34525 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](SortMerger)add error throwing mechanism [doris]
TomaYoko commented on PR #34457: URL: https://github.com/apache/doris/pull/34457#issuecomment-2099896517 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve] Make the schema change memory space adaptive [doris]
github-actions[bot] commented on PR #34515: URL: https://github.com/apache/doris/pull/34515#issuecomment-2099898657 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve] Make the schema change memory space adaptive [doris]
github-actions[bot] commented on PR #34515: URL: https://github.com/apache/doris/pull/34515#issuecomment-2099898708 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improvement](spill) fuzzy spill and fix bugs [doris]
jacktengg commented on PR #33291: URL: https://github.com/apache/doris/pull/33291#issuecomment-2099907433 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refine](dataque) use lock free queue to impl dataqueue [doris]
Mryange commented on PR #34468: URL: https://github.com/apache/doris/pull/34468#issuecomment-2099907628 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](file-scanner) print current path when encountering error (#34365) [doris]
doris-robot commented on PR #34524: URL: https://github.com/apache/doris/pull/34524#issuecomment-2099911423 TeamCity be ut coverage result: Function Coverage: 37.80% (8076/21365) Line Coverage: 29.45% (65932/223889) Region Coverage: 28.91% (33938/117386) Branch Coverage: 24.77% (17417/70310) Coverage Report: http://coverage.selectdb-in.cc/coverage/dc88f0a0a35e74ad45a474eef1bbc12bee676638_dc88f0a0a35e74ad45a474eef1bbc12bee676638/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](file-scanner) print current path when encountering error (#34365) [doris]
doris-robot commented on PR #34524: URL: https://github.com/apache/doris/pull/34524#issuecomment-2099917075 TPC-DS: Total hot run time: 201998 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit dc88f0a0a35e74ad45a474eef1bbc12bee676638, data reload: false query1 933 381 372 372 query2 6525264027092640 query3 6924200 194 194 query4 20142 17925 17879 17879 query5 19721 656664686468 query6 293 228 240 228 query7 4158316 295 295 query8 286 235 247 235 query9 3146268726412641 query10 412 288 314 288 query11 11367 10798 10636 10636 query12 118 71 74 71 query13 5577684 657 657 query14 17798 13704 13442 13442 query15 357 218 225 218 query16 6444267 254 254 query17 17351445869 869 query18 2309404 399 399 query19 202 144 142 142 query20 76 72 72 72 query21 185 100 90 90 query22 5297520351085108 query23 32580 31808 31878 31808 query24 6905654464496449 query25 513 422 396 396 query26 520 160 158 158 query27 1816288 292 288 query28 6213241923272327 query29 2844267526362636 query30 242 166 160 160 query31 914 714 725 714 query32 65 64 55 55 query33 404 248 263 248 query34 861 468 488 468 query35 1130907 941 907 query36 1207109212351092 query37 88 60 57 57 query38 3140293929182918 query39 1350131313231313 query40 202 92 89 89 query41 39 34 34 34 query42 85 84 85 84 query43 782 735 659 659 query44 1141731 730 730 query45 242 222 226 222 query46 1224943 947 943 query47 1719163317221633 query48 1020708 707 707 query49 606 362 372 362 query50 888 653 577 577 query51 4747463846624638 query52 89 71 74 71 query53 431 317 323 317 query54 2668248424512451 query55 85 73 77 73 query56 199 211 211 211 query57 1164110311051103 query58 212 205 188 188 query59 4102420839373937 query60 210 196 214 196 query61 90 83 82 82 query62 900 521 461 461 query63 467 338 332 332 query64 2484155614541454 query65 3639353035663530 query66 768 367 371 367 query67 15935 14927 15810 14927 query68 10839 667 671 667 query69 572 337 345 337 query70 2002136612521252 query71 417 291 309 291 query72 6437344334313431 query73 746 316 311 311 query74 6391583458725834 query75 5550377937283728 query76 6951110211491102 query77 1193241 243 241 query78 12514 11765 12170 11765 query79 9131648 659 648 query80 850 389 394 389 query81 457 228 227 227 query82 870 94 95 94 query83 158 129 129 129 query84 264 71 66 66 query85 856 289 292 289 query86 310 295 300 295 query87 3249305130083008 query88 4414228822782278 query89 476 272 273 272 query90 1931207 205 205 query91 163 143 133 133 query92 54 50 52 50 query93 7043584 582 582 query94 652 204 200 200 query95 1081104410421042 query96 650 324 318 318 query97 6502636462736273 query98 179 170 161 161 query99 3129810 820 810 Total cold run time: 316966 ms Total hot run time: 201998 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [opt](Nereids) support where, group by, having, order by clause without from clause in query statement [doris]
doris-robot commented on PR #34475: URL: https://github.com/apache/doris/pull/34475#issuecomment-2099917390 TPC-H: Total hot run time: 50259 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit c2118a67caa9b79d8f42cf38d7503c985905319c, data reload: false -- Round 1 -- q1 17560 440942774277 q2 2023157 144 144 q3 10266 188119081881 q4 10106 125313401253 q5 8512396139423942 q6 238 128 127 127 q7 2032159115991591 q8 9253273027172717 q9 11031 11529 10886 10886 q10 8592348235303482 q11 415 249 251 249 q12 468 300 305 300 q13 18366 397040053970 q14 358 327 318 318 q15 514 470 458 458 q16 695 575 586 575 q17 1139956 953 953 q18 7281679168366791 q19 1720159815571557 q20 557 305 305 305 q21 4432413340974097 q22 490 386 399 386 Total cold run time: 116048 ms Total hot run time: 50259 ms - Round 2, with runtime_filter_mode=off - q1 4306427543034275 q2 323 224 227 224 q3 4161417541314131 q4 2734272127352721 q5 7187714571847145 q6 239 120 117 117 q7 3292282528722825 q8 4331442544454425 q9 17393 17046 17069 17046 q10 4259423442464234 q11 737 654 647 647 q12 1032863 869 863 q13 4468371636973697 q14 452 421 423 421 q15 499 466 449 449 q16 724 674 680 674 q17 3793383138433831 q18 8818870387808703 q19 1710169516011601 q20 2359214721372137 q21 8486841584848415 q22 1005935 931 931 Total cold run time: 82308 ms Total hot run time: 79512 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [Improve] Estimated column reader memory to control segment cache [doris]
Lchangliang opened a new pull request, #34526: URL: https://github.com/apache/doris/pull/34526 ## Proposed changes Issue Number: close #xxx If a segment has many columns, it will use a lot more memory than a segment with few columns, but it is not observable at present. So we estimated column reader memory to to control the memory. ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
doris-robot commented on PR #34526: URL: https://github.com/apache/doris/pull/34526#issuecomment-2099923883 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Lchangliang commented on PR #34526: URL: https://github.com/apache/doris/pull/34526#issuecomment-2099924079 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](file-scanner) print current path when encountering error (#34365) [doris]
doris-robot commented on PR #34524: URL: https://github.com/apache/doris/pull/34524#issuecomment-2099926749 ClickBench: Total hot run time: 30.86 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit dc88f0a0a35e74ad45a474eef1bbc12bee676638, data reload: false query1 0.020.020.02 query2 0.070.020.02 query3 0.240.050.04 query4 1.810.060.06 query5 0.530.520.52 query6 1.230.610.62 query7 0.010.010.01 query8 0.030.020.02 query9 0.530.500.48 query10 0.550.540.55 query11 0.120.090.08 query12 0.120.100.09 query13 0.620.610.62 query14 0.790.780.77 query15 0.770.760.77 query16 0.370.370.38 query17 0.971.010.98 query18 0.230.250.23 query19 1.941.851.86 query20 0.020.010.01 query21 15.47 0.560.56 query22 2.151.912.58 query23 17.19 1.000.80 query24 5.091.250.91 query25 0.380.130.04 query26 0.610.160.16 query27 0.060.040.04 query28 7.870.740.70 query29 12.64 2.482.40 query30 0.610.550.49 query31 2.820.390.37 query32 3.410.490.49 query33 3.073.123.07 query34 15.25 4.794.78 query35 4.854.864.83 query36 1.041.001.03 query37 0.050.040.04 query38 0.040.020.03 query39 0.020.010.01 query40 0.160.140.14 query41 0.060.010.01 query42 0.030.010.01 query43 0.020.010.02 Total cold run time: 103.86 s Total hot run time: 30.86 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](test) remove enable_nereids_planner in regression cases (part 2) [doris]
github-actions[bot] commented on PR #34490: URL: https://github.com/apache/doris/pull/34490#issuecomment-2099927841 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [pipeline](refactor) remove pipeline task state [doris]
Gabriel39 opened a new pull request, #34527: URL: https://github.com/apache/doris/pull/34527 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipeline](refactor) remove pipeline task state [doris]
Gabriel39 commented on PR #34527: URL: https://github.com/apache/doris/pull/34527#issuecomment-2099927726 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipeline](refactor) remove pipeline task state [doris]
doris-robot commented on PR #34527: URL: https://github.com/apache/doris/pull/34527#issuecomment-2099929925 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](file-scanner) print current path when encountering error (#34365) [doris]
doris-robot commented on PR #34524: URL: https://github.com/apache/doris/pull/34524#issuecomment-2099931889 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit dc88f0a0a35e74ad45a474eef1bbc12bee676638 with default session variables Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 21.3 seconds inserted 1000 Rows, about 469K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
github-actions[bot] commented on PR #34526: URL: https://github.com/apache/doris/pull/34526#issuecomment-2099934088 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
xinyiZzz commented on PR #34526: URL: https://github.com/apache/doris/pull/34526#issuecomment-2099935198  ``` auto* lru_handle = LRUCachePolicy::insert( key.encode(), &value, 1, value.segment->meta_mem_usage(), CachePriority::NORMAL); ``` change to ``` auto* lru_handle = LRUCachePolicy::insert( key.encode(), &value, value.segment->meta_mem_usage(), value.segment->meta_mem_usage(), CachePriority::NORMAL); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](Nereids) support where, group by, having, order by clause without from clause in query statement [doris]
doris-robot commented on PR #34475: URL: https://github.com/apache/doris/pull/34475#issuecomment-2099938717 TPC-DS: Total hot run time: 202612 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit c2118a67caa9b79d8f42cf38d7503c985905319c, data reload: false query1 924 393 369 369 query2 6524274627352735 query3 6915205 198 198 query4 19939 18003 17854 17854 query5 19736 652964856485 query6 287 221 235 221 query7 4150311 316 311 query8 242 256 219 219 query9 3081265925892589 query10 415 290 299 290 query11 11271 10702 10717 10702 query12 122 72 77 72 query13 5573697 690 690 query14 17747 13301 13411 13301 query15 377 227 229 227 query16 6473270 260 260 query17 17201474874 874 query18 2306405 400 400 query19 210 145 155 145 query20 79 80 80 80 query21 193 91 91 91 query22 5254499850204998 query23 32732 32014 31873 31873 query24 6935646464356435 query25 521 415 425 415 query26 538 161 159 159 query27 1872294 301 294 query28 6198241423502350 query29 2999262526762625 query30 238 161 162 161 query31 918 728 750 728 query32 67 64 62 62 query33 401 250 254 250 query34 856 477 479 477 query35 1156933 964 933 query36 1313104711371047 query37 88 58 60 58 query38 3052292529452925 query39 1356131013331310 query40 206 95 93 93 query41 41 43 37 37 query42 83 84 82 82 query43 776 662 759 662 query44 1124739 739 739 query45 244 233 224 224 query46 1231966 969 966 query47 1769166816381638 query48 1006707 703 703 query49 615 372 370 370 query50 863 637 629 629 query51 4744463846724638 query52 87 86 87 86 query53 448 320 318 318 query54 2654243124202420 query55 92 87 81 81 query56 224 215 206 206 query57 1206112710511051 query58 203 212 210 210 query59 4222409940564056 query60 217 188 209 188 query61 89 84 84 84 query62 831 492 464 464 query63 476 347 343 343 query64 2538147614691469 query65 3631360435393539 query66 822 383 382 382 query67 15849 15141 15393 15141 query68 9620650 648 648 query69 592 354 351 351 query70 1745142313761376 query71 417 313 313 313 query72 6497343234413432 query73 730 328 325 325 query74 6368588358905883 query75 5312366337553663 query76 6048113612071136 query77 1017260 253 253 query78 12946 11886 11660 11660 query79 8210644 632 632 query80 1401382 389 382 query81 495 238 229 229 query82 1571100 97 97 query83 178 131 129 129 query84 255 69 71 69 query85 850 297 297 297 query86 342 310 297 297 query87 3171306130103010 query88 4842233323132313 query89 392 273 276 273 query90 1943210 205 205 query91 165 132 164 132 query92 57 56 52 52 query93 6332565 609 565 query94 691 202 203 202 query95 104810351035 query96 645 332 321 321 query97 6599642063526352 query98 195 176 164 164 query99 2843784 953 784 Total cold run time: 314649 ms Total hot run time: 202612 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
xinyiZzz commented on code in PR #34526:
URL: https://github.com/apache/doris/pull/34526#discussion_r1593531683
##
be/src/olap/rowset/segment_v2/segment.cpp:
##
@@ -408,6 +408,7 @@ Status Segment::_create_column_readers(const
SegmentFooterPB& footer) {
RETURN_IF_ERROR(ColumnReader::create(opts,
footer.columns(iter->second), footer.num_rows(),
_file_reader, &reader));
_column_readers.emplace(column.unique_id(), std::move(reader));
+_meta_mem_usage += config::estimated_mem_per_column_reader;
Review Comment:
in past, `_meta_mem_usage` only tracking some indexes memory,
now, add column reader memory
will there be other memory in segment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipeline](refactor) remove pipeline task state [doris]
github-actions[bot] commented on PR #34527: URL: https://github.com/apache/doris/pull/34527#issuecomment-2099939849 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](regression-test) modify a key type from BIGINT/LARGEINT to other type [doris]
dataroaring merged PR #34436: URL: https://github.com/apache/doris/pull/34436 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (f1b6bc4da99 -> adad523d7b3)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
from f1b6bc4da99 [fix](merge-on-write) remove some CHECKs in
Tablet::revise_tablet_meta (#31268)
add adad523d7b3 [enhancement](regression-test) modify a key type from
BIGINT/LARGEINT to other type (#34436)
No new revisions were added by this update.
Summary of changes:
.../test_unique_schema_key_change_add.out | 111
...ut => test_unique_schema_key_change_modify.out} | 130 +---
.../test_unique_schema_key_change_add.groovy | 285 +
...=> test_unique_schema_key_change_modify.groovy} | 696 +
4 files changed, 842 insertions(+), 380 deletions(-)
create mode 100644
regression-test/data/schema_change_p0/test_unique_schema_key_change_add.out
copy
regression-test/data/schema_change_p0/{test_unique_model_schema_key_change.out
=> test_unique_schema_key_change_modify.out} (53%)
create mode 100644
regression-test/suites/schema_change_p0/test_unique_schema_key_change_add.groovy
rename
regression-test/suites/schema_change_p0/{test_unique_model_schema_key_change.groovy
=> test_unique_schema_key_change_modify.groovy} (66%)
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](test) remove enable_nereids_planner in regression cases (part 2) [doris]
github-actions[bot] commented on PR #34490: URL: https://github.com/apache/doris/pull/34490#issuecomment-2099942204 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](test) remove enable_nereids_planner in regression cases (part 2) [doris]
morrySnow merged PR #34490: URL: https://github.com/apache/doris/pull/34490 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated (adad523d7b3 -> a22b4f4d379)
This is an automated email from the ASF dual-hosted git repository. morrysnow pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/doris.git from adad523d7b3 [enhancement](regression-test) modify a key type from BIGINT/LARGEINT to other type (#34436) add a22b4f4d379 [WIP](test) remove enable_nereids_planner in regression cases (part 2) (#34490) No new revisions were added by this update. Summary of changes: .../test_bloom_filter_is_not_null.out | 10 .../data/correctness/test_cast_as_decimalv3.out| 6 -- .../test_from_millisecond_microsecond.out | 9 --- regression-test/data/correctness/test_in_null.out | 14 - .../data/correctness/test_str_to_date.out | 34 ++- .../correctness/test_time_diff_microseconds.out| 2 +- .../data/correctness/test_time_function.out| 48 --- regression-test/data/correctness_p0/test_avg.out | 5 +- .../data/correctness_p0/test_cast_null.out | 3 - .../data/correctness_p0/test_constant_having.out | 2 - .../correctness_p0/test_div_decimal_overflow.out | 5 +- .../data/correctness_p0/test_function_conjunct.out | 2 - regression-test/data/ddl_p0/test_create_view.out | 2 +- .../data/performance_p0/redundant_conjuncts.out| 29 +++-- .../test_bloom_filter_is_not_null.groovy | 12 +--- .../correctness/test_cast_as_decimalv3.groovy | 14 + .../correctness/test_date_function_const.groovy| 5 +- .../test_from_millisecond_microsecond.groovy | 21 ++- .../suites/correctness/test_in_null.groovy | 15 + .../suites/correctness/test_str_to_date.groovy | 28 ++--- .../correctness/test_time_diff_microseconds.groovy | 5 +- .../suites/correctness/test_time_function.groovy | 68 +- .../suites/correctness/test_timev2_fold.groovy | 6 +- .../correctness_p0/test_agg_materialize.groovy | 1 - .../suites/correctness_p0/test_avg.groovy | 6 -- .../suites/correctness_p0/test_bit_function.groovy | 2 - .../correctness_p0/test_bit_shift_lagency.groovy | 4 +- .../correctness_p0/test_bucket_shuffle_join.groovy | 11 ++-- .../correctness_p0/test_cast_date_decimal.groovy | 4 -- .../suites/correctness_p0/test_cast_in.groovy | 3 - .../suites/correctness_p0/test_cast_null.groovy| 17 -- .../correctness_p0/test_constant_having.groovy | 21 --- .../suites/correctness_p0/test_ctas_mv.groovy | 1 - .../test_div_decimal_overflow.groovy | 9 --- .../correctness_p0/test_function_conjunct.groovy | 5 -- .../suites/correctness_p0/test_implict_cast.groovy | 3 +- .../test_inlineview_with_lateralview.groovy| 1 - .../test_inlineview_with_project.groovy| 1 - .../test_inlineview_with_window_function.groovy| 1 - .../test_join_with_projection.groovy | 1 - .../suites/correctness_p0/test_mv_case.groovy | 1 - .../suites/correctness_p0/test_time_round.groovy | 4 +- .../datatype_p0/complex_types/test_map.groovy | 1 - .../test_complextype_information_schema.groovy | 1 - ..._nested_types_insert_into_with_agg_table.groovy | 3 +- ..._nested_types_insert_into_with_dup_table.groovy | 1 - ...sted_types_insert_into_with_unique_table.groovy | 3 +- .../suites/ddl_p0/test_create_view.groovy | 1 - regression-test/suites/ddl_p0/test_ctas.groovy | 6 -- ...urrent_timestamp_as_column_default_value.groovy | 4 -- .../suites/delete_p0/test_delete_on_value.groovy | 3 - .../cache/parse_sql_from_sql_cache.groovy | 10 .../nereids_p0/create_table/test_ctas.groovy | 2 - .../suites/nereids_p0/datatype/test_cast.groovy| 7 +-- regression-test/suites/nereids_p0/load.groovy | 3 - .../test_string_function_like.groovy | 11 .../table_function/explode_map.groovy | 7 --- .../suites/nereids_p0/tablesample.groovy | 4 +- .../nereids_p0/test_filter_pushdown_set.groovy | 3 +- .../mv/newMv/dup_mv_bm_hash.groovy | 8 --- .../nereids_syntax_p0/mv/newMv/dup_mv_year.groovy | 8 --- .../mv/newMv/sum_devide_count.groovy | 6 -- .../performance_p0/redundant_conjuncts.groovy | 2 +- 63 files changed, 63 insertions(+), 472 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](default value): enable HLL_EMPTY_DEFAULT_VALUE [doris]
my-vegetable-has-exploded commented on PR #34447: URL: https://github.com/apache/doris/pull/34447#issuecomment-2099942731 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [docs](fix) add some parameters for stream load and some tips for tvf and `show profile`. [doris-website]
yagagagaga opened a new pull request, #623: URL: https://github.com/apache/doris-website/pull/623 1. streamload add description of "memtable_on_sink_node:true" 2. `show query profile` has removed since 2.1.1 3. active_queries and workload_groups has been moved to system table since 2.1.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)Mtmv rollup [doris]
seawinde commented on PR #31812: URL: https://github.com/apache/doris/pull/31812#issuecomment-2099948682 check null partition is correct in the partition roll up scene -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](Nereids) support where, group by, having, order by clause without from clause in query statement [doris]
doris-robot commented on PR #34475: URL: https://github.com/apache/doris/pull/34475#issuecomment-2099949151 ClickBench: Total hot run time: 30.46 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit c2118a67caa9b79d8f42cf38d7503c985905319c, data reload: false query1 0.030.020.02 query2 0.070.020.03 query3 0.250.040.05 query4 1.780.070.06 query5 0.530.530.52 query6 1.240.630.62 query7 0.010.010.01 query8 0.030.030.02 query9 0.540.470.47 query10 0.550.520.54 query11 0.120.080.09 query12 0.110.090.09 query13 0.620.610.62 query14 0.780.790.78 query15 0.780.770.76 query16 0.390.370.36 query17 1.001.000.96 query18 0.230.250.24 query19 1.901.851.84 query20 0.010.010.02 query21 15.47 0.540.56 query22 2.042.181.32 query23 16.86 1.090.96 query24 7.941.461.01 query25 0.400.090.04 query26 0.880.150.14 query27 0.040.030.04 query28 4.810.740.78 query29 12.60 2.282.34 query30 0.600.520.51 query31 2.810.390.37 query32 3.400.500.49 query33 3.053.103.05 query34 15.24 4.824.82 query35 4.884.864.86 query36 1.041.021.02 query37 0.060.050.04 query38 0.040.020.02 query39 0.020.020.01 query40 0.160.140.14 query41 0.070.010.01 query42 0.020.010.01 query43 0.020.020.02 Total cold run time: 103.42 s Total hot run time: 30.46 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)Mtmv rollup [doris]
github-actions[bot] commented on PR #31812: URL: https://github.com/apache/doris/pull/31812#issuecomment-2099951187 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](hive)fix error for writing to hive [doris]
github-actions[bot] commented on PR #34520: URL: https://github.com/apache/doris/pull/34520#issuecomment-2099951563 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Lchangliang commented on code in PR #34526:
URL: https://github.com/apache/doris/pull/34526#discussion_r1593541497
##
be/src/olap/rowset/segment_v2/segment.cpp:
##
@@ -408,6 +408,7 @@ Status Segment::_create_column_readers(const
SegmentFooterPB& footer) {
RETURN_IF_ERROR(ColumnReader::create(opts,
footer.columns(iter->second), footer.num_rows(),
_file_reader, &reader));
_column_readers.emplace(column.unique_id(), std::move(reader));
+_meta_mem_usage += config::estimated_mem_per_column_reader;
Review Comment:
It still has some unique_ptr/shared_ptr object and the object
`SubcolumnColumnReaders`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](Nereids) support where, group by, having, order by clause without from clause in query statement [doris]
doris-robot commented on PR #34475: URL: https://github.com/apache/doris/pull/34475#issuecomment-2099955287 Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' ``` Load test result on commit c2118a67caa9b79d8f42cf38d7503c985905319c with default session variables Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s Insert into select: 21.4 seconds inserted 1000 Rows, about 467K ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Revert dragonbox related [doris]
doris-robot commented on PR #34528: URL: https://github.com/apache/doris/pull/34528#issuecomment-2099961941 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] Revert dragonbox related [doris]
zhiqiang- opened a new pull request, #34528: URL: https://github.com/apache/doris/pull/34528 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Revert dragonbox related [doris]
zhiqiang- commented on PR #34528: URL: https://github.com/apache/doris/pull/34528#issuecomment-2099965184 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](session) fix select * from variables system table [doris]
nextdreamblue opened a new pull request, #34529: URL: https://github.com/apache/doris/pull/34529 ## Proposed changes Issue Number: close #xxx after #23017, information_schema.session_variables and information_schema.global_variables had 2 new columns, but SchemaVariablesScanner just 2 columns. when select * from *_variables,well get 'no match column for this column' error ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](session) fix select * from variables system table [doris]
doris-robot commented on PR #34529: URL: https://github.com/apache/doris/pull/34529#issuecomment-2099965622 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](session) fix select * from variables system table [doris]
nextdreamblue commented on PR #34529: URL: https://github.com/apache/doris/pull/34529#issuecomment-2099965953 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](hive)fix error for writing to hive [doris]
morningman commented on PR #34520: URL: https://github.com/apache/doris/pull/34520#issuecomment-2099968793 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](hive)fix error for writing to hive [doris]
github-actions[bot] commented on PR #34520: URL: https://github.com/apache/doris/pull/34520#issuecomment-2099969738 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](hive)fix error for writing to hive [doris]
github-actions[bot] commented on PR #34520: URL: https://github.com/apache/doris/pull/34520#issuecomment-2099969801 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] DorisSource createEnumerator error when has filter condition(pushDown) [doris-flink-connector]
Tyxuani commented on issue #376: URL: https://github.com/apache/doris-flink-connector/issues/376#issuecomment-2099970143 Resolved this by myself: `tinyint` `is_aa` can't use to condition `is_aa = 1` , need to define as `int`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] Revert dragonbox related [doris]
github-actions[bot] commented on PR #34528: URL: https://github.com/apache/doris/pull/34528#issuecomment-2099971852 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](session) fix select * from variables system table [doris]
github-actions[bot] commented on PR #34529: URL: https://github.com/apache/doris/pull/34529#issuecomment-2099974453 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [pipeline](fix) Logging blocking dependency if task is not cancelled [doris]
doris-robot commented on PR #34502: URL: https://github.com/apache/doris/pull/34502#issuecomment-2099973491 TeamCity be ut coverage result: Function Coverage: 35.68% (8981/25170) Line Coverage: 27.34% (74187/271342) Region Coverage: 26.57% (38348/144304) Branch Coverage: 23.40% (19559/83598) Coverage Report: http://coverage.selectdb-in.cc/coverage/478cb61937381cb7fde72d574080e6b02fd5eabf_478cb61937381cb7fde72d574080e6b02fd5eabf/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](session) fix select * from variables system table [doris]
zhiqiang- commented on PR #34529: URL: https://github.com/apache/doris/pull/34529#issuecomment-2099981057 I have tried to do similar fix in https://github.com/apache/doris/pull/30998, but not merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] test [doris]
zzzxl1993 commented on PR #34394: URL: https://github.com/apache/doris/pull/34394#issuecomment-2099986574 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] test [doris]
zzzxl1993 commented on PR #34394: URL: https://github.com/apache/doris/pull/34394#issuecomment-2099986747 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.1](cherry-pick)fix refresh failed when not use db before create MTMV (#34… [doris]
yiguolei merged PR #34522: URL: https://github.com/apache/doris/pull/34522 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] 查询和写入报错（doris-2.1.1-rc05） [doris]
lexluo09 commented on issue #34308: URL: https://github.com/apache/doris/issues/34308#issuecomment-2099987180 好的，谢谢 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (e085f75a434 -> 202cdb2744d)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from e085f75a434 [opt](file-scanner) print current path when encountering error (#34365) (#34523) add 202cdb2744d [fix](mtmv)fix refresh failed when not use db before create MTMV (#34431) (#34522) No new revisions were added by this update. Summary of changes: .../apache/doris/job/extensions/mtmv/MTMVTask.java | 14 +++- .../java/org/apache/doris/mtmv/MTMVPlanUtil.java | 43 -- .../trees/plans/visitor/TableCollector.java| 14 +--- .../doris/cluster/DecommissionBackendTest.java | 4 +- .../suites/mtmv_p0/test_env_db_dropped_mtmv.groovy | 93 ++ 5 files changed, 144 insertions(+), 24 deletions(-) create mode 100644 regression-test/suites/mtmv_p0/test_env_db_dropped_mtmv.groovy - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Bug] 查询和写入报错（doris-2.1.1-rc05） [doris]
lexluo09 closed issue #34308: [Bug] 查询和写入报错（doris-2.1.1-rc05） URL: https://github.com/apache/doris/issues/34308 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](catalog) refactor the schema cache for external table [doris]
github-actions[bot] commented on PR #34517: URL: https://github.com/apache/doris/pull/34517#issuecomment-2099988092 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] test [doris]
zzzxl1993 commented on PR #34394: URL: https://github.com/apache/doris/pull/34394#issuecomment-204721 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](local exchange) fix bug of local exchange free blocks [doris]
jacktengg opened a new pull request, #34530: URL: https://github.com/apache/doris/pull/34530 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](local exchange) fix bug of local exchange free blocks [doris]
doris-robot commented on PR #34530: URL: https://github.com/apache/doris/pull/34530#issuecomment-214394 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](local exchange) fix bug of local exchange free blocks [doris]
jacktengg commented on PR #34530: URL: https://github.com/apache/doris/pull/34530#issuecomment-215730 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] test [doris]
zzzxl1993 commented on PR #34394: URL: https://github.com/apache/doris/pull/34394#issuecomment-2100015530 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](type) change some type inheritance relationship and remove useless interface [doris]
morrySnow commented on PR #34463: URL: https://github.com/apache/doris/pull/34463#issuecomment-2100015722 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](local exchange) fix bug of local exchange free blocks [doris]
github-actions[bot] commented on PR #34530: URL: https://github.com/apache/doris/pull/34530#issuecomment-2100015229 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [improve] Make the schema change memory space adaptive [doris]
xiaokang merged PR #34515: URL: https://github.com/apache/doris/pull/34515 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [improve] Make the schema change memory space adaptive #34350 (#34515)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 778a8877108 [improve] Make the schema change memory space adaptive
#34350 (#34515)
778a8877108 is described below
commit 778a8877108027b5877a1e23d1831d97a247a27e
Author: Lightman <31928846+lchangli...@users.noreply.github.com>
AuthorDate: Wed May 8 16:20:59 2024 +0800
[improve] Make the schema change memory space adaptive #34350 (#34515)
---
be/src/common/config.cpp | 3 +++
be/src/common/config.h| 3 +++
be/src/olap/schema_change.h | 5 -
be/src/olap/storage_engine.cpp| 9 +
be/src/olap/storage_engine.h | 4
be/src/olap/task/engine_alter_tablet_task.cpp | 9 -
be/src/olap/task/engine_index_change_task.cpp | 5 -
7 files changed, 35 insertions(+), 3 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 82b85b44421..34181f4d256 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -85,6 +85,9 @@ DEFINE_String(mem_limit, "80%");
// Soft memory limit as a fraction of hard memory limit.
DEFINE_Double(soft_mem_limit_frac, "0.9");
+// Schema change memory limit as a fraction of soft memory limit.
+DEFINE_Double(schema_change_mem_limit_frac, "0.6");
+
// Many modern allocators (for example, tcmalloc) do not do a mremap for
// realloc, even in case of large enough chunks of memory. Although this allows
// you to increase performance and reduce memory consumption during realloc.
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 007c5082135..7665b4866dd 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -120,6 +120,9 @@ DECLARE_String(mem_limit);
// Soft memory limit as a fraction of hard memory limit.
DECLARE_Double(soft_mem_limit_frac);
+// Schema change memory limit as a fraction of soft memory limit.
+DECLARE_Double(schema_change_mem_limit_frac);
+
// Many modern allocators (for example) do not do a mremap for
// realloc, even in case of large enough chunks of memory. Although this allows
// you to increase performance and reduce memory consumption during realloc.
diff --git a/be/src/olap/schema_change.h b/be/src/olap/schema_change.h
index 60942f671b0..c13f543d726 100644
--- a/be/src/olap/schema_change.h
+++ b/be/src/olap/schema_change.h
@@ -43,6 +43,7 @@
#include "olap/rowset/rowset_reader.h"
#include "olap/rowset/rowset_writer.h"
#include "olap/rowset/segment_v2/inverted_index_writer.h"
+#include "olap/storage_engine.h"
#include "olap/tablet.h"
#include "olap/tablet_schema.h"
#include "runtime/descriptors.h"
@@ -231,7 +232,9 @@ public:
bool sc_sorting,
bool sc_directly) {
if (sc_sorting) {
return std::make_unique(
-changer,
config::memory_limitation_per_thread_for_schema_change_bytes);
+changer, ExecEnv::GetInstance()
+ ->storage_engine()
+
->memory_limitation_bytes_per_thread_for_schema_change());
}
if (sc_directly) {
diff --git a/be/src/olap/storage_engine.cpp b/be/src/olap/storage_engine.cpp
index 9ff97db8291..0e633c81ef4 100644
--- a/be/src/olap/storage_engine.cpp
+++ b/be/src/olap/storage_engine.cpp
@@ -73,6 +73,7 @@
#include "runtime/memory/mem_tracker.h"
#include "runtime/stream_load/stream_load_recorder.h"
#include "util/doris_metrics.h"
+#include "util/mem_info.h"
#include "util/metrics.h"
#include "util/spinlock.h"
#include "util/stopwatch.hpp"
@@ -141,6 +142,9 @@ StorageEngine::StorageEngine(const EngineOptions& options)
});
_broken_paths = options.broken_paths;
+
+_memory_limitation_bytes_for_schema_change =
+static_cast(MemInfo::soft_mem_limit() *
config::schema_change_mem_limit_frac);
}
StorageEngine::~StorageEngine() {
@@ -169,6 +173,11 @@ StorageEngine::~StorageEngine() {
_s_instance = nullptr;
}
+int64_t StorageEngine::memory_limitation_bytes_per_thread_for_schema_change()
const {
+return std::max(_memory_limitation_bytes_for_schema_change /
config::alter_tablet_worker_count,
+
config::memory_limitation_per_thread_for_schema_change_bytes);
+}
+
Status StorageEngine::load_data_dirs(const std::vector& data_dirs) {
std::vector threads;
std::vector results(data_dirs.size());
diff --git a/be/src/olap/storage_engine.h b/be/src/olap/storage_engine.h
index 6c1f18b4c20..e450c504121 100644
--- a/be/src/olap/storage_engine.h
+++ b/be/src/olap/storage_engine.h
@@ -233,6 +233,8 @@ public:
std::set get_broken_paths() { return _broken_paths; }
+int64_t memory_limitation_bytes_per_thread_for_schema_change() const;
+
p
Re: [PR] [improve] Make the schema change memory space adaptive [doris]
xiaokang commented on PR #34515: URL: https://github.com/apache/doris/pull/34515#issuecomment-2100020028 #34350 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](file-scanner) print current path when encountering error (#34365) [doris]
xiaokang merged PR #34524: URL: https://github.com/apache/doris/pull/34524 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [opt](file-scanner) print current path when encountering error (#34365) (#34524)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new a73792e62f9 [opt](file-scanner) print current path when encountering
error (#34365) (#34524)
a73792e62f9 is described below
commit a73792e62f994e01a4fb1e2708eb40ebff31d82d
Author: Mingyu Chen
AuthorDate: Wed May 8 16:21:41 2024 +0800
[opt](file-scanner) print current path when encountering error (#34365)
(#34524)
---
be/src/vec/exec/scan/vfile_scanner.cpp | 11 ++-
be/src/vec/exec/scan/vfile_scanner.h | 2 ++
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/be/src/vec/exec/scan/vfile_scanner.cpp
b/be/src/vec/exec/scan/vfile_scanner.cpp
index dc4d643a80d..3a6c9ff7eb7 100644
--- a/be/src/vec/exec/scan/vfile_scanner.cpp
+++ b/be/src/vec/exec/scan/vfile_scanner.cpp
@@ -240,6 +240,15 @@ Status VFileScanner::open(RuntimeState* state) {
return Status::OK();
}
+Status VFileScanner::_get_block_impl(RuntimeState* state, Block* block, bool*
eof) {
+Status st = _get_block_wrapped(state, block, eof);
+if (!st.ok()) {
+// add cur path in error msg for easy debugging
+return std::move(st.prepend("cur path: " +
get_current_scan_range_name() + ". "));
+}
+return st;
+}
+
// For query:
// [exist cols] [non-exist cols] [col from
path] input output
// A BC D E
@@ -259,7 +268,7 @@ Status VFileScanner::open(RuntimeState* state) {
// _fill_columns_from_path - -- - x
x -
// _fill_missing_columns- -- x -
x -
// _convert_to_output_block - -- - -
- x
-Status VFileScanner::_get_block_impl(RuntimeState* state, Block* block, bool*
eof) {
+Status VFileScanner::_get_block_wrapped(RuntimeState* state, Block* block,
bool* eof) {
do {
if (_cur_reader == nullptr || _cur_reader_eof) {
RETURN_IF_ERROR(_get_next_reader());
diff --git a/be/src/vec/exec/scan/vfile_scanner.h
b/be/src/vec/exec/scan/vfile_scanner.h
index 5261337f000..0344b04fc1c 100644
--- a/be/src/vec/exec/scan/vfile_scanner.h
+++ b/be/src/vec/exec/scan/vfile_scanner.h
@@ -84,6 +84,8 @@ public:
protected:
Status _get_block_impl(RuntimeState* state, Block* block, bool* eof)
override;
+Status _get_block_wrapped(RuntimeState* state, Block* block, bool* eof);
+
Status _get_next_reader();
// TODO: cast input block columns type to string.
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.0][Improve](config)The stream_load label length is changed to a configurable [doris]
xiaokang merged PR #34458: URL: https://github.com/apache/doris/pull/34458 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.0 updated: [Improve](config)The stream_load label length is changed to a configurable #33745 (#34458)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 9061b8142ad [Improve](config)The stream_load label length is changed
to a configurable #33745 (#34458)
9061b8142ad is described below
commit 9061b8142adb7fd184784fc1665be009bc91c4f3
Author: wudongliang <46414265+donglian...@users.noreply.github.com>
AuthorDate: Wed May 8 16:22:30 2024 +0800
[Improve](config)The stream_load label length is changed to a configurable
#33745 (#34458)
---
fe/fe-common/src/main/java/org/apache/doris/common/Config.java | 4
fe/fe-core/src/main/java/org/apache/doris/common/FeNameFormat.java | 2 +-
2 files changed, 5 insertions(+), 1 deletion(-)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index dccfa777668..61692471964 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -2354,4 +2354,8 @@ public class Config extends ConfigBase {
})
public static boolean enable_proxy_protocol = false;
+@ConfField(description = {"Stream_Load 导入时，label 被限制的最大长度",
+"Stream_Load When importing, the maximum length of label is
limited"})
+public static int label_regex_length = 128;
+
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/FeNameFormat.java
b/fe/fe-core/src/main/java/org/apache/doris/common/FeNameFormat.java
index 5a10bce7954..e21cbe7b112 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/FeNameFormat.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/FeNameFormat.java
@@ -28,7 +28,7 @@ import org.apache.doris.qe.VariableMgr;
import com.google.common.base.Strings;
public class FeNameFormat {
-private static final String LABEL_REGEX = "^[-_A-Za-z0-9:]{1,128}$";
+private static final String LABEL_REGEX = "^[-_A-Za-z0-9:]{1," +
Config.label_regex_length + "}$";
private static final String COMMON_NAME_REGEX =
"^[a-zA-Z][a-zA-Z0-9-_]{0,63}$";
private static final String UNDERSCORE_COMMON_NAME_REGEX =
"^[_a-zA-Z][a-zA-Z0-9-_]{0,63}$";
private static final String TABLE_NAME_REGEX = "^[a-zA-Z][a-zA-Z0-9-_]*$";
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](compaction) generate single compaction and cumu(base) compaction tablets simultaneously. [doris]
csun5285 commented on PR #34315: URL: https://github.com/apache/doris/pull/34315#issuecomment-2100023358 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](hive)fix error for writing to hive [doris]
doris-robot commented on PR #34520: URL: https://github.com/apache/doris/pull/34520#issuecomment-2100024147 TeamCity be ut coverage result: Function Coverage: 35.68% (8981/25170) Line Coverage: 27.33% (74186/271410) Region Coverage: 26.57% (38346/144339) Branch Coverage: 23.39% (19560/83624) Coverage Report: http://coverage.selectdb-in.cc/coverage/1ee2a0bc91ffe4b945fa45a7f64e6059fd47fd8c_1ee2a0bc91ffe4b945fa45a7f64e6059fd47fd8c/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](local exchange) fix bug of local exchange free blocks [doris]
github-actions[bot] commented on PR #34530: URL: https://github.com/apache/doris/pull/34530#issuecomment-2100024178 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](local exchange) fix bug of local exchange free blocks [doris]
github-actions[bot] commented on PR #34530: URL: https://github.com/apache/doris/pull/34530#issuecomment-2100024224 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](local exchange) fix bug of local exchange free blocks [doris]
jacktengg opened a new pull request, #34531: URL: https://github.com/apache/doris/pull/34531 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix][cdc] fix uid conflicts during multi-database synchronization. [doris-flink-connector]
vinlee19 opened a new pull request, #382: URL: https://github.com/apache/doris-flink-connector/pull/382 # Proposed changes When dealing with multiple databases in MySQL, it's common to have tables with identical schemas spread across different databases. If you try to synchronize these databases to Doris using the provided shell script, you might encounter the following error: ```bash rg.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Hash collision on user-specified ID "customer_1". Most likely cause is a non-unique ID. Please check that all IDs specified via `uid(String)` are unique. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245) at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095) at org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) at org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157) Caused by: java.lang.IllegalArgumentException: Hash collision on user-specified ID "customer_1". Most likely cause is a non-unique ID. Please check that all IDs specified via `uid(String)` are unique. at org.apache.flink.streaming.api.graph.StreamGraphHasherV2.generateNodeHash(StreamGraphHasherV2.java:185) at org.apache.flink.streaming.api.graph.StreamGraphHasherV2.traverseStreamGraphAndGenerateHashes(StreamGraphHasherV2.java:110) at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:246) at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:160) at org.apache.flink.streaming.api.graph.StreamGraph.getJobGraph(StreamGraph.java:1024) at org.apache.flink.client.StreamGraphTranslator.translateToJobGraph(StreamGraphTranslator.java:56) at org.apache.flink.client.FlinkPipelineTranslationUtil.getJobGraph(FlinkPipelineTranslationUtil.java:43) at org.apache.flink.client.deployment.executors.PipelineExecutorUtils.getJobGraph(PipelineExecutorUtils.java:61) at org.apache.flink.client.deployment.executors.AbstractSessionClusterExecutor.execute(AbstractSessionClusterExecutor.java:72) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2197) at org.apache.flink.client.program.StreamContextEnvironment.executeAsync(StreamContextEnvironment.java:189) at org.apache.flink.client.program.StreamContextEnvironment.execute(StreamContextEnvironment.java:118) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:2058) at org.apache.doris.flink.tools.cdc.CdcTools.syncDatabase(CdcTools.java:151) at org.apache.doris.flink.tools.cdc.CdcTools.createMySQLSyncDatabase(CdcTools.java:74) at org.apache.doris.flink.tools.cdc.CdcTools.main(CdcTools.java:51) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ``` This PR aims to address this issue. Issue Number: close #xxx ## Problem Summary: Describe the overview of changes. ## Checklist(Required) 1. Does it affect the original behavior: (Yes/No/I Don't know) 3. Has unit tests been added: (Yes/No/No Need) 4. Has document been added or modified: (Yes/No/No Need) 5. Does it need to update dependencies: (Yes/No) 6. Are there any changes that cannot be rolled back: (Yes/No) ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache
Re: [PR] [fix](local exchange) fix bug of local exchange free blocks [doris]
doris-robot commented on PR #34531: URL: https://github.com/apache/doris/pull/34531#issuecomment-2100025924 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [test](fix) Fix test check fail when test nested mv hit [doris]
morrySnow merged PR #34293: URL: https://github.com/apache/doris/pull/34293 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [test](fix) Fix test check fail when test nested mv hit (#34293)
This is an automated email from the ASF dual-hosted git repository.
morrysnow pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d20b18f222e [test](fix) Fix test check fail when test nested mv hit
(#34293)
d20b18f222e is described below
commit d20b18f222ea1a60c88b43a51bdea529f53f506b
Author: seawinde <149132972+seawi...@users.noreply.github.com>
AuthorDate: Wed May 8 16:25:33 2024 +0800
[test](fix) Fix test check fail when test nested mv hit (#34293)
if mv3 is def as following:
select c1, c2, c3 from t1;
mv4 is def as following:
select c1, c2 from mv3;
when query is
select c1, c2 from t1;
the mv3 and mv4 both can be rewritten successfully
---
.../suites/nereids_rules_p0/mv/nested_mtmv/nested_mtmv.groovy | 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git
a/regression-test/suites/nereids_rules_p0/mv/nested_mtmv/nested_mtmv.groovy
b/regression-test/suites/nereids_rules_p0/mv/nested_mtmv/nested_mtmv.groovy
index 881d91d0a45..ce64cc21bf1 100644
--- a/regression-test/suites/nereids_rules_p0/mv/nested_mtmv/nested_mtmv.groovy
+++ b/regression-test/suites/nereids_rules_p0/mv/nested_mtmv/nested_mtmv.groovy
@@ -279,7 +279,11 @@ suite("nested_mtmv") {
"""
explain {
sql("${query_stmt_2}")
-contains "${mv_level4_name}(${mv_level4_name})"
+check {result ->
+// both mv_level4_name and mv_level3_name can be rewritten
successfully
+result.contains("${mv_level4_name}(${mv_level4_name})")
+|| result.contains("${mv_level3_name}(${mv_level3_name})")
+}
}
compare_res(query_stmt_2 + " order by 1,2,3,4,5,6,7")
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](local exchange) fix bug of local exchange free blocks [doris]
github-actions[bot] commented on PR #34531: URL: https://github.com/apache/doris/pull/34531#issuecomment-2100030142 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](local exchange) fix bug of local exchange free blocks [doris]
github-actions[bot] commented on PR #34531: URL: https://github.com/apache/doris/pull/34531#issuecomment-2100030198 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](local exchange) fix bug of local exchange free blocks [doris]
yiguolei merged PR #34531: URL: https://github.com/apache/doris/pull/34531 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated: [fix](local exchange) fix bug of local exchange free blocks (#34531)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.1 by this push:
new 06612b9be31 [fix](local exchange) fix bug of local exchange free
blocks (#34531)
06612b9be31 is described below
commit 06612b9be31fdb37f769efcbb7a39e58e3b1fc85
Author: TengJianPing <18241664+jackte...@users.noreply.github.com>
AuthorDate: Wed May 8 16:26:51 2024 +0800
[fix](local exchange) fix bug of local exchange free blocks (#34531)
---
be/src/pipeline/pipeline_x/local_exchange/local_exchanger.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/pipeline/pipeline_x/local_exchange/local_exchanger.cpp
b/be/src/pipeline/pipeline_x/local_exchange/local_exchanger.cpp
index 0837a1212b9..71e427f8b25 100644
--- a/be/src/pipeline/pipeline_x/local_exchange/local_exchanger.cpp
+++ b/be/src/pipeline/pipeline_x/local_exchange/local_exchanger.cpp
@@ -104,7 +104,7 @@ Status ShuffleExchanger::_split_rows(RuntimeState* state,
const uint32_t* __rest
vectorized::Block data_block;
std::shared_ptr new_block_wrapper;
-if (_free_blocks.try_enqueue(data_block)) {
+if (_free_blocks.try_dequeue(data_block)) {
new_block_wrapper =
ShuffleBlockWrapper::create_shared(std::move(data_block));
} else {
new_block_wrapper =
ShuffleBlockWrapper::create_shared(block->clone_empty());
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [branch-2.0](tablet invert index) fix tablet invert index leaky caused by auto partition #33973 [doris]
yujun777 commented on PR #34514: URL: https://github.com/apache/doris/pull/34514#issuecomment-2100031669 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](local exchange) fix bug of local exchange free blocks [doris]
github-actions[bot] commented on PR #34531: URL: https://github.com/apache/doris/pull/34531#issuecomment-2100045231 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Lchangliang commented on PR #34526: URL: https://github.com/apache/doris/pull/34526#issuecomment-2100045752 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
