Re: [PR] [opt](spill) handle oom exception in spill tasks [doris]
github-actions[bot] commented on code in PR #35025:
URL: https://github.com/apache/doris/pull/35025#discussion_r1605689692
##
be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:
##
@@ -153,43 +153,64 @@ Status
PartitionedHashJoinProbeLocalState::close(RuntimeState* state) {
return Status::OK();
}
-Status PartitionedHashJoinProbeLocalState::spill_build_block(RuntimeState*
state,
- uint32_t
partition_index) {
-auto& partitioned_build_blocks = _shared_state->partitioned_build_blocks;
-auto& mutable_block = partitioned_build_blocks[partition_index];
-if (!mutable_block ||
-mutable_block->allocated_bytes() <
vectorized::SpillStream::MIN_SPILL_WRITE_BATCH_MEM) {
---_spilling_task_count;
-return Status::OK();
-}
-
-auto& build_spilling_stream =
_shared_state->spilled_streams[partition_index];
-if (!build_spilling_stream) {
-
RETURN_IF_ERROR(ExecEnv::GetInstance()->spill_stream_mgr()->register_spill_stream(
-state, build_spilling_stream, print_id(state->query_id()),
"hash_build_sink",
-_parent->id(), std::numeric_limits::max(),
-std::numeric_limits::max(), _runtime_profile.get()));
-RETURN_IF_ERROR(build_spilling_stream->prepare_spill());
-
build_spilling_stream->set_write_counters(_spill_serialize_block_timer,
_spill_block_count,
- _spill_data_size,
_spill_write_disk_timer,
- _spill_write_wait_io_timer);
-}
-
+Status PartitionedHashJoinProbeLocalState::spill_probe_blocks(RuntimeState*
state) {
Review Comment:
warning: function 'spill_probe_blocks' has cognitive complexity of 59
(threshold 50) [readability-function-cognitive-complexity]
```cpp
Status PartitionedHashJoinProbeLocalState::spill_probe_blocks(RuntimeState*
state) {
^
```
Additional context
**be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:169:**
nesting level increased to 1
```cpp
auto spill_func = [query_id, state, submit_timer, this] {
^
```
**be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:174:** +2,
including nesting penalty of 1, nesting level increased to 2
```cpp
for (uint32_t partition_index = 0; partition_index !=
p._partition_count;
^
```
**be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:178:** +3,
including nesting penalty of 2, nesting level increased to 3
```cpp
if (partitioned_block && partitioned_block->allocated_bytes() >=
^
```
**be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:178:** +1
```cpp
if (partitioned_block && partitioned_block->allocated_bytes() >=
^
```
**be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:185:** +3,
including nesting penalty of 2, nesting level increased to 3
```cpp
if (!spilling_stream) {
^
```
**be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:186:** +4,
including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(ExecEnv::GetInstance()->spill_stream_mgr()->register_spill_stream(
^
```
**be/src/common/status.h:614:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do {\
^
```
**be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:186:** +5,
including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(ExecEnv::GetInstance()->spill_stream_mgr()->register_spill_stream(
^
```
**be/src/common/status.h:616:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:190:** +4,
including nesting penalty of 3, nesting level increased to 4
```cpp
RETURN_IF_ERROR(spilling_stream->prepare_spill());
^
```
**be/src/common/status.h:614:** expanded from macro 'RETURN_IF_ERROR'
```cpp
do {\
^
```
**be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:190:** +5,
including nesting penalty of 4, nesting level increased to 5
```cpp
RETURN_IF_ERROR(spilling_stream->prepare_spill());
^
```
**be/src/common/status.h:616:** expanded from macro 'RETURN_IF_ERROR'
```cpp
if (UNLIKELY(!_status_.ok())) { \
^
```
**be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:197:** +3,
including nesting penalty of 2, nesting level increased
Re: [PR] [opt](spill) handle oom exception in spill tasks [doris]
doris-robot commented on PR #35025: URL: https://github.com/apache/doris/pull/35025#issuecomment-2118675408 TeamCity be ut coverage result: Function Coverage: 35.69% (9008/25241) Line Coverage: 27.36% (74501/272321) Region Coverage: 26.60% (38534/144887) Branch Coverage: 23.43% (19657/83900) Coverage Report: http://coverage.selectdb-in.cc/coverage/9b5f312ab0e958c43aeb91b9009ed94ef9d1062c_9b5f312ab0e958c43aeb91b9009ed94ef9d1062c/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory [doris]
yiguolei commented on code in PR #34985:
URL: https://github.com/apache/doris/pull/34985#discussion_r1605693725
##
be/src/runtime/memory/global_memory_arbitrator.h:
##
@@ -0,0 +1,133 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "util/mem_info.h"
+
+namespace doris {
+
+class GlobalMemoryArbitrator {
+public:
+static inline int64_t proc_mem_no_allocator_cache() {
+return _s_proc_mem_no_allocator_cache.load(std::memory_order_relaxed);
+}
+static inline int64_t proc_mem_corrected() {
Review Comment:
add comment to explain the function。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [chore](fuzzy) add more fuzzy config [doris]
dataroaring opened a new pull request, #35036: URL: https://github.com/apache/doris/pull/35036 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](fuzzy) add more fuzzy config [doris]
dataroaring commented on PR #35036: URL: https://github.com/apache/doris/pull/35036#issuecomment-2118678480 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](fuzzy) add more fuzzy config [doris]
doris-robot commented on PR #35036: URL: https://github.com/apache/doris/pull/35036#issuecomment-2118678461 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](fuzzy) add more fuzzy config [doris]
github-actions[bot] commented on PR #35036: URL: https://github.com/apache/doris/pull/35036#issuecomment-2118679862 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory [doris]
yiguolei commented on code in PR #34985:
URL: https://github.com/apache/doris/pull/34985#discussion_r1605696566
##
be/src/runtime/memory/thread_mem_tracker_mgr.h:
##
@@ -164,16 +172,35 @@ inline bool
ThreadMemTrackerMgr::push_consumer_tracker(MemTracker* tracker) {
}
_consumer_tracker_stack.push_back(tracker);
tracker->release(_untracked_mem);
+tracker->consume(_reserved_mem); // consume in advance
return true;
}
inline void ThreadMemTrackerMgr::pop_consumer_tracker() {
DCHECK(!_consumer_tracker_stack.empty());
_consumer_tracker_stack.back()->consume(_untracked_mem);
+_consumer_tracker_stack.back()->release(_reserved_mem);
_consumer_tracker_stack.pop_back();
}
inline void ThreadMemTrackerMgr::consume(int64_t size, int
skip_large_memory_check) {
+if (_reserved_mem != 0) {
+if (_reserved_mem >= size) {
Review Comment:
这里要加注释
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [chore](fuzzy) add more fuzzy config [doris]
doris-robot commented on PR #35036: URL: https://github.com/apache/doris/pull/35036#issuecomment-2118683253 TeamCity be ut coverage result: Function Coverage: 35.70% (9006/25228) Line Coverage: 27.36% (74477/272255) Region Coverage: 26.60% (38512/144808) Branch Coverage: 23.43% (19656/83894) Coverage Report: http://coverage.selectdb-in.cc/coverage/797b944b5e0fa833279b504834490528f2f9bc5c_797b944b5e0fa833279b504834490528f2f9bc5c/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory [doris]
yiguolei commented on code in PR #34985:
URL: https://github.com/apache/doris/pull/34985#discussion_r1605696893
##
be/src/runtime/memory/thread_mem_tracker_mgr.h:
##
@@ -209,14 +236,58 @@ inline void ThreadMemTrackerMgr::flush_untracked_mem() {
_stop_consume = true;
DCHECK(_limiter_tracker_raw);
-old_untracked_mem = _untracked_mem;
-if (_count_scope_mem) _scope_mem += _untracked_mem;
-_limiter_tracker_raw->consume(old_untracked_mem);
-for (auto tracker : _consumer_tracker_stack) {
-tracker->consume(old_untracked_mem);
+_old_untracked_mem = _untracked_mem;
+if (_count_scope_mem) {
+_scope_mem += _untracked_mem;
+}
+_limiter_tracker_raw->consume(_old_untracked_mem);
+for (auto* tracker : _consumer_tracker_stack) {
+tracker->consume(_old_untracked_mem);
}
-_untracked_mem -= old_untracked_mem;
+_untracked_mem -= _old_untracked_mem;
_stop_consume = false;
}
+inline bool ThreadMemTrackerMgr::try_reserve(int64_t size, bool
force_tracker_overcommit) {
Review Comment:
why need overcommit?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [feature](datatype) add BE config to allow zero date [doris]
doris-robot commented on PR #34961: URL: https://github.com/apache/doris/pull/34961#issuecomment-2118684650 TeamCity be ut coverage result: Function Coverage: 35.71% (9009/25228) Line Coverage: 27.37% (74523/272254) Region Coverage: 26.63% (38556/144805) Branch Coverage: 23.45% (19675/83894) Coverage Report: http://coverage.selectdb-in.cc/coverage/73b64113eaa9aa7687cf9ff43ae6175f17c1a68d_73b64113eaa9aa7687cf9ff43ae6175f17c1a68d/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory [doris]
yiguolei commented on code in PR #34985:
URL: https://github.com/apache/doris/pull/34985#discussion_r1605698375
##
be/src/common/daemon.cpp:
##
@@ -247,33 +248,35 @@ void Daemon::memory_gc_thread() {
continue;
}
auto sys_mem_available = doris::MemInfo::sys_mem_available();
-auto proc_mem_no_allocator_cache =
doris::MemInfo::proc_mem_no_allocator_cache();
+auto proc_mem_corrected =
doris::GlobalMemoryArbitrator::proc_mem_corrected();
// GC excess memory for resource groups that not enable overcommit
-auto tg_free_mem = doris::MemInfo::tg_not_enable_overcommit_group_gc();
+auto tg_free_mem = doris::MemInfo::tg_disable_overcommit_group_gc();
sys_mem_available += tg_free_mem;
-proc_mem_no_allocator_cache -= tg_free_mem;
+proc_mem_corrected -= tg_free_mem;
Review Comment:
proc_mem_corrected 这个名字太难懂了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [load](cloud) Add FE config for streamload redirect policy: streamload_redirect_policy [doris]
doris-robot commented on PR #35031: URL: https://github.com/apache/doris/pull/35031#issuecomment-2118685699 TPC-DS: Total hot run time: 188837 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 8ff9938559110442ebd6ae9cb132b67fb8a9d876, data reload: false query1 936 396 363 363 query2 6296248223882388 query3 6647215 220 215 query4 23479 21970 21995 21970 query5 3763430 413 413 query6 265 189 177 177 query7 4541291 296 291 query8 240 185 194 185 query9 8553243224622432 query10 408 244 246 244 query11 15292 14808 14764 14764 query12 120 96 91 91 query13 1597362 363 362 query14 10386 780177447744 query15 258 166 173 166 query16 8096261 263 261 query17 1683591 560 560 query18 2077272 273 272 query19 193 155 150 150 query20 90 83 86 83 query21 195 129 122 122 query22 5178490448564856 query23 34307 33512 33436 33436 query24 10876 294329352935 query25 618 407 394 394 query26 1127159 155 155 query27 2897320 317 317 query28 7554202120052005 query29 879 629 624 624 query30 275 178 171 171 query31 958 743 743 743 query32 97 51 52 51 query33 743 249 248 248 query34 1026485 472 472 query35 795 664 664 664 query36 1075942 938 938 query37 134 70 70 70 query38 2901279927702770 query39 1631157615911576 query40 198 128 127 127 query41 46 44 46 44 query42 103 104 103 103 query43 591 532 532 532 query44 1172724 736 724 query45 266 259 255 255 query46 1082728 768 728 query47 1973189718711871 query48 386 302 295 295 query49 905 407 395 395 query50 756 396 379 379 query51 6956673168366731 query52 107 89 92 89 query53 352 292 278 278 query54 850 427 430 427 query55 78 74 77 74 query56 247 225 228 225 query57 1234115311701153 query58 226 204 214 204 query59 3385317332513173 query60 273 243 245 243 query61 108 100 101 100 query62 673 463 473 463 query63 311 285 284 284 query64 8862741074017401 query65 3124310730633063 query66 908 330 333 330 query67 15585 15112 15005 15005 query68 10063 545 537 537 query69 602 313 304 304 query70 137810441044 query71 524 267 265 265 query72 8667255924182418 query73 1580319 314 314 query74 6676615360896089 query75 5505263926312631 query76 54681025947 947 query77 670 267 260 260 query78 10925 10187 10174 10174 query79 8748522 521 521 query80 1257436 422 422 query81 511 243 247 243 query82 235 91 94 91 query83 194 163 163 163 query84 264 85 81 81 query85 969 274 262 262 query86 351 299 317 299 query87 3259311030573057 query88 4731231823212318 query89 516 398 378 378 query90 2121189 189 189 query91 125 96 97 96 query92 57 47 49 47 query93 6053503 491 491 query94 1307179 180 179 query95 382 299 292 292 query96 602 266 269 266 query97 3154296030172960 query98 236 226 218 218 query99 1150910 907 907 Total cold run time: 307223 ms Total hot run time: 188837 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th
Re: [PR] [Enhancement](Nereids) Nereids supports group_commit with insert [doris]
zzzs commented on PR #32523: URL: https://github.com/apache/doris/pull/32523#issuecomment-2118685907 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [load](cloud) Add FE config for streamload redirect policy: streamload_redirect_policy [doris]
doris-robot commented on PR #35031: URL: https://github.com/apache/doris/pull/35031#issuecomment-2118687332 ClickBench: Total hot run time: 30.7 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 8ff9938559110442ebd6ae9cb132b67fb8a9d876, data reload: false query1 0.040.030.04 query2 0.070.040.04 query3 0.230.050.05 query4 1.660.090.10 query5 0.500.490.54 query6 1.130.720.72 query7 0.020.010.01 query8 0.060.040.04 query9 0.540.490.48 query10 0.530.540.54 query11 0.150.120.12 query12 0.140.120.12 query13 0.600.590.60 query14 0.770.790.78 query15 0.830.820.80 query16 0.360.370.37 query17 1.021.021.02 query18 0.200.240.25 query19 1.801.651.73 query20 0.020.010.01 query21 15.66 0.700.67 query22 4.297.182.06 query23 18.31 1.391.25 query24 1.300.410.25 query25 0.150.080.07 query26 0.260.160.17 query27 0.080.070.07 query28 13.30 1.011.00 query29 13.75 3.293.29 query30 0.230.050.06 query31 2.860.410.40 query32 3.260.470.46 query33 2.792.842.86 query34 17.09 4.404.43 query35 4.474.524.62 query36 0.640.450.49 query37 0.180.160.14 query38 0.170.150.14 query39 0.050.030.04 query40 0.160.140.14 query41 0.080.050.05 query42 0.050.040.05 query43 0.040.030.03 Total cold run time: 109.84 s Total hot run time: 30.7 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [refactor](status) refactor querycontext and runtime state status [doris]
doris-robot commented on PR #35035: URL: https://github.com/apache/doris/pull/35035#issuecomment-2118690325 TeamCity be ut coverage result: Function Coverage: 35.72% (9010/25226) Line Coverage: 27.37% (74499/272214) Region Coverage: 26.61% (38524/144787) Branch Coverage: 23.44% (19659/83882) Coverage Report: http://coverage.selectdb-in.cc/coverage/d662369717dc8181fe19b0f74aa4433c0e2387d6_d662369717dc8181fe19b0f74aa4433c0e2387d6/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [fix](agg) memory leak issue in agg operator [doris]
mrhhsg opened a new pull request, #35037: URL: https://github.com/apache/doris/pull/35037 ## Proposed changes Issue Number: close #xxx ## Further comments If this is a relatively large or complex change, kick off the discussion at [d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](agg) memory leak issue in agg operator [doris]
doris-robot commented on PR #35037: URL: https://github.com/apache/doris/pull/35037#issuecomment-2118751394 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](agg) memory leak issue in agg operator [doris]
mrhhsg commented on PR #35037: URL: https://github.com/apache/doris/pull/35037#issuecomment-2118751614 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](agg) memory leak issue in agg operator [doris]
mrhhsg commented on PR #35037: URL: https://github.com/apache/doris/pull/35037#issuecomment-2118752430 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](agg) memory leak issue in agg operator [doris]
github-actions[bot] commented on PR #35037: URL: https://github.com/apache/doris/pull/35037#issuecomment-2118753064 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (b76cfcd007b -> 80dd027ce2e)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
from b76cfcd007b [refactor](mtmv) Materialization context and mtmv
decoupling (#34093) (#34916)
new 691f3c5ee7a [Performance](Variant) Improve load performance for
variant type (#33890)
new 7e967e53b83 Fix failed p2 hive statistics case. (#34663)
new 876248aa4e3 [fix](function) json_object can not input null value
(#34591)
new 6f91e9cc4dc [fix](test) fix s3 load test failed (#34671)
new e2614d453a8 [case](regression) Add hdfs backup restore case (#34716)
new b3b848f862d [feature](Nereids): eliminate useless project (#34611)
new 5012ddd87a2 [fix](Nereids) fix sql cache return old value when
truncate partition (#34698)
new c71d0b6b223 [fix](Nereids) cast from json should always nullable
(#34707)
new 46bf43130f7 [test](case) error format case in test_query_json_object
(#34722)
new 1545d96617c [WIP](test) remove enable_nereids_planner in regression
cases (part 4) (#34642)
new b6409f55841 [improvement](inverted index) Disable the use of skipping
write index on load (#34719)
new 80dd027ce2e [opt](join) For left semi/anti join without mark join
conjunct and without other conjucnts, stop probing after matching one row
(#34703)
The 12 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/vec/columns/column_object.cpp | 189 +++---
be/src/vec/columns/column_object.h | 28 +-
be/src/vec/common/hash_table/join_hash_table.h | 31 +-
be/src/vec/common/schema_util.cpp | 57 +--
be/src/vec/common/schema_util.h| 4 +-
be/src/vec/core/field.h| 5 +
.../vec/exec/join/process_hash_table_probe_impl.h | 3 +-
be/src/vec/functions/function_json.cpp | 24 +-
be/src/vec/json/parse2column.cpp | 22 +-
.../main/java/org/apache/doris/common/Config.java | 8 +-
.../analysis/ModifyTablePropertiesClause.java | 5 +
.../main/java/org/apache/doris/catalog/Env.java| 4 +-
.../doris/common/NereidsSqlCacheManager.java | 38 +-
.../apache/doris/common/util/PropertyAnalyzer.java | 3 +-
.../apache/doris/datasource/InternalCatalog.java | 3 +-
.../processor/post/ColumnPruningPostProcessor.java | 11 +-
.../doris/nereids/trees/expressions/Cast.java | 2 +
.../doris/nereids/trees/plans/algebra/Project.java | 10 +
.../trees/plans/logical/LogicalProject.java| 10 -
.../org/apache/doris/qe/cache/CacheAnalyzer.java | 2 +-
.../correctness_p0/test_group_having_alias.out | 2 -
.../test_array_functions_array_intersect_sort.out | 110 +-
.../data/javaudf_p0/test_javaudf_with_decimal.out | 8 +-
...hdfs.groovy => test_backup_restore_hdfs.groovy} | 34 +-
.../suites/correctness_p0/test_cast_decimal.groovy | 14 -
.../correctness_p0/test_group_having_alias.groovy | 20 --
...est_array_functions_array_intersect_sort.groovy | 97 +
.../suites/datatype_p0/scalar_types/load.groovy| 2 +-
...l_tvf_with_complex_type_insertinto_doris.groovy | 1 -
.../hive/test_hive_partition_column_analyze.groovy | 390 +++--
.../hive/test_hive_partition_statistic.groovy | 3 +
.../tvf/test_tvf_view_count_p2.groovy | 16 +-
.../javaudf_p0/test_javaudf_with_decimal.groovy| 37 +-
.../test_s3_load_with_load_parallelism.groovy | 4 +-
.../cache/parse_sql_from_sql_cache.groovy | 38 +-
.../suites/query_p0/cast/test_cast.groovy | 18 +
.../json_function/test_query_json_object.groovy| 4 +
...st_array_with_single_row_max_than_uint16.groovy | 1 -
.../suites/variant_p2/performance.groovy | 36 ++
39 files changed, 673 insertions(+), 621 deletions(-)
copy
regression-test/suites/backup_restore/{test_backup_restore_with_hdfs.groovy =>
test_backup_restore_hdfs.groovy} (76%)
create mode 100644 regression-test/suites/variant_p2/performance.groovy
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 07/12: [fix](Nereids) fix sql cache return old value when truncate partition (#34698)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5012ddd87a2a671bc2aed465732adc2def929993
Author: 924060929 <924060...@qq.com>
AuthorDate: Mon May 13 12:05:22 2024 +0800
[fix](Nereids) fix sql cache return old value when truncate partition
(#34698)
1. fix sql cache return old value when truncate partition
2. use expire_sql_cache_in_fe_second to control the expire time of the sql
cache which in the NereidsSqlCacheManager
---
.../main/java/org/apache/doris/common/Config.java | 8 -
.../main/java/org/apache/doris/catalog/Env.java| 4 +--
.../doris/common/NereidsSqlCacheManager.java | 38 +++---
.../org/apache/doris/qe/cache/CacheAnalyzer.java | 2 +-
.../cache/parse_sql_from_sql_cache.groovy | 38 +-
5 files changed, 65 insertions(+), 25 deletions(-)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 6fc20578ec0..d44ff4a0dd4 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -1306,12 +1306,18 @@ public class Config extends ConfigBase {
* Minimum interval between last version when caching results,
* This parameter distinguishes between offline and real-time updates
*/
+@ConfField(mutable = true, masterOnly = false)
+public static int cache_last_version_interval_second = 30;
+
+/**
+ * Expire sql sql in frontend time
+ */
@ConfField(
mutable = true,
masterOnly = false,
callbackClassString =
"org.apache.doris.common.NereidsSqlCacheManager$UpdateConfig"
)
-public static int cache_last_version_interval_second = 30;
+public static int expire_sql_cache_in_fe_second = 300;
/**
* Set the maximum number of rows that can be cached
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
index 4839769e0f8..d27ae147323 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
@@ -771,9 +771,7 @@ public class Env {
this.mtmvService = new MTMVService();
this.insertOverwriteManager = new InsertOverwriteManager();
this.dnsCache = new DNSCache();
-this.sqlCacheManager = new NereidsSqlCacheManager(
-Config.sql_cache_manage_num,
Config.cache_last_version_interval_second
-);
+this.sqlCacheManager = new NereidsSqlCacheManager();
}
public static void destroyCheckpoint() {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/NereidsSqlCacheManager.java
b/fe/fe-core/src/main/java/org/apache/doris/common/NereidsSqlCacheManager.java
index 8989375c07f..cf6280650f0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/NereidsSqlCacheManager.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/NereidsSqlCacheManager.java
@@ -74,8 +74,11 @@ public class NereidsSqlCacheManager {
// value: SqlCacheContext
private volatile Cache sqlCaches;
-public NereidsSqlCacheManager(int sqlCacheNum, long cacheIntervalSeconds) {
-sqlCaches = buildSqlCaches(sqlCacheNum, cacheIntervalSeconds);
+public NereidsSqlCacheManager() {
+sqlCaches = buildSqlCaches(
+Config.sql_cache_manage_num,
+Config.expire_sql_cache_in_fe_second
+);
}
public static synchronized void updateConfig() {
@@ -90,22 +93,24 @@ public class NereidsSqlCacheManager {
Cache sqlCaches = buildSqlCaches(
Config.sql_cache_manage_num,
-Config.cache_last_version_interval_second
+Config.expire_sql_cache_in_fe_second
);
sqlCaches.putAll(sqlCacheManager.sqlCaches.asMap());
sqlCacheManager.sqlCaches = sqlCaches;
}
-private static Cache buildSqlCaches(int
sqlCacheNum, long cacheIntervalSeconds) {
-sqlCacheNum = sqlCacheNum < 0 ? 100 : sqlCacheNum;
-cacheIntervalSeconds = cacheIntervalSeconds < 0 ? 30 :
cacheIntervalSeconds;
-
-return Caffeine.newBuilder()
-.maximumSize(sqlCacheNum)
-.expireAfterAccess(Duration.ofSeconds(cacheIntervalSeconds))
+private static Cache buildSqlCaches(int
sqlCacheNum, long expireAfterAccessSeconds) {
+Caffeine cacheBuilder = Caffeine.newBuilder()
// auto evict cache when jvm memory too low
-.softValues()
-.build();
+.softValues();
+if (sqlCacheNum > 0) {
+cacheBuilder = cacheBuilder.maximumSize(sqlCacheNum);
+}
+if (expireAfterAccessSeconds > 0) {
+
(doris) 06/12: [feature](Nereids): eliminate useless project (#34611)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit b3b848f862d857d5fa439ff808c59aabd56e9cf7
Author: jakevin
AuthorDate: Mon May 13 10:17:29 2024 +0800
[feature](Nereids): eliminate useless project (#34611)
---
.../nereids/processor/post/ColumnPruningPostProcessor.java| 11 +--
.../org/apache/doris/nereids/trees/plans/algebra/Project.java | 10 ++
.../doris/nereids/trees/plans/logical/LogicalProject.java | 10 --
3 files changed, 19 insertions(+), 12 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ColumnPruningPostProcessor.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ColumnPruningPostProcessor.java
index ff362b7bc9e..e8c3cf4a6ac 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ColumnPruningPostProcessor.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/ColumnPruningPostProcessor.java
@@ -40,9 +40,16 @@ import java.util.stream.Collectors;
})
public class ColumnPruningPostProcessor extends PlanPostProcessor {
@Override
-public PhysicalProject visitPhysicalProject(PhysicalProject project, CascadesContext ctx) {
+public Plan visitPhysicalProject(PhysicalProject project,
CascadesContext ctx) {
Plan child = project.child();
Plan newChild = child.accept(this, ctx);
+if (project.isAllSlots()) {
+Set projects =
project.getProjects().stream().map(Slot.class::cast).collect(Collectors.toSet());
+Set outputSet = newChild.getOutputSet();
+if (outputSet.equals(projects)) {
+return ((AbstractPhysicalPlan)
newChild).copyStatsAndGroupIdFrom(project);
+}
+}
if (newChild instanceof AbstractPhysicalJoin) {
AbstractPhysicalJoin join =
(AbstractPhysicalJoin) newChild;
Plan left = join.left();
@@ -91,7 +98,7 @@ public class ColumnPruningPostProcessor extends
PlanPostProcessor {
}
if (newLeft != left || newRight != right) {
-return (PhysicalProject)
project.withChildren(join.withChildren(newLeft, newRight));
+return project.withChildren(join.withChildren(newLeft,
newRight));
} else {
return project;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/algebra/Project.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/algebra/Project.java
index 7fa62f7628f..15b095bee3b 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/algebra/Project.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/algebra/Project.java
@@ -117,4 +117,14 @@ public interface Project {
return expr;
});
}
+
+/** isAllSlots */
+default boolean isAllSlots() {
+for (NamedExpression project : getProjects()) {
+if (!project.isSlot()) {
+return false;
+}
+}
+return true;
+}
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalProject.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalProject.java
index de23bc6f5b0..24d938636aa 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalProject.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalProject.java
@@ -114,16 +114,6 @@ public class LogicalProject
extends LogicalUnary computeOutput() {
Builder slots =
ImmutableList.builderWithExpectedSize(projects.size());
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 12/12: [opt](join) For left semi/anti join without mark join conjunct and without other conjucnts, stop probing after matching one row (#34703)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 80dd027ce2e28a6fb5c6c1b9eaaf7c7ff5f2a068
Author: Jerry Hu
AuthorDate: Mon May 13 23:46:27 2024 +0800
[opt](join) For left semi/anti join without mark join conjunct and without
other conjucnts, stop probing after matching one row (#34703)
---
be/src/vec/common/hash_table/join_hash_table.h | 31 +++---
.../vec/exec/join/process_hash_table_probe_impl.h | 3 ++-
2 files changed, 24 insertions(+), 10 deletions(-)
diff --git a/be/src/vec/common/hash_table/join_hash_table.h
b/be/src/vec/common/hash_table/join_hash_table.h
index 10ca6c9b2dd..a869ad419ad 100644
--- a/be/src/vec/common/hash_table/join_hash_table.h
+++ b/be/src/vec/common/hash_table/join_hash_table.h
@@ -89,7 +89,7 @@ public:
auto find_batch(const Key* __restrict keys, const uint32_t* __restrict
build_idx_map,
int probe_idx, uint32_t build_idx, int probe_rows,
uint32_t* __restrict probe_idxs, bool& probe_visited,
-uint32_t* __restrict build_idxs) {
+uint32_t* __restrict build_idxs, bool
has_mark_join_conjunct = false) {
if constexpr (JoinOpType == TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN ||
JoinOpType == TJoinOp::NULL_AWARE_LEFT_SEMI_JOIN) {
if (_empty_build_side) {
@@ -100,12 +100,25 @@ public:
if constexpr (with_other_conjuncts ||
(is_mark_join && JoinOpType !=
TJoinOp::RIGHT_SEMI_JOIN)) {
-constexpr bool null_aware_without_other_conjuncts =
-(JoinOpType == TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN ||
- JoinOpType == TJoinOp::NULL_AWARE_LEFT_SEMI_JOIN) &&
-!with_other_conjuncts;
-return _find_batch_conjunct(
+if constexpr (!with_other_conjuncts) {
+constexpr bool is_null_aware_join =
+JoinOpType == TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN ||
+JoinOpType == TJoinOp::NULL_AWARE_LEFT_SEMI_JOIN;
+constexpr bool is_left_half_join = JoinOpType ==
TJoinOp::LEFT_SEMI_JOIN ||
+ JoinOpType ==
TJoinOp::LEFT_ANTI_JOIN;
+
+/// For null aware join or left half(semi/anti) join without
other conjuncts and without
+/// mark join conjunct.
+/// If one row on probe side has one match in build side, we
should stop searching the
+/// hash table for this row.
+if (is_null_aware_join || (is_left_half_join &&
!has_mark_join_conjunct)) {
+return _find_batch_conjunct(
+keys, build_idx_map, probe_idx, build_idx,
probe_rows, probe_idxs,
+build_idxs);
+}
+}
+
+return _find_batch_conjunct(
keys, build_idx_map, probe_idx, build_idx, probe_rows,
probe_idxs, build_idxs);
}
@@ -314,7 +327,7 @@ private:
return std::tuple {probe_idx, 0U, matched_cnt};
}
-template
+template
auto _find_batch_conjunct(const Key* __restrict keys, const uint32_t*
__restrict build_idx_map,
int probe_idx, uint32_t build_idx, int
probe_rows,
uint32_t* __restrict probe_idxs, uint32_t*
__restrict build_idxs) {
@@ -345,7 +358,7 @@ private:
probe_idxs[matched_cnt] = probe_idx;
matched_cnt++;
-if constexpr (null_aware_without_other_conjuncts) {
+if constexpr (only_need_to_match_one) {
build_idx = 0;
break;
}
diff --git a/be/src/vec/exec/join/process_hash_table_probe_impl.h
b/be/src/vec/exec/join/process_hash_table_probe_impl.h
index b4212405aed..9a15df637aa 100644
--- a/be/src/vec/exec/join/process_hash_table_probe_impl.h
+++ b/be/src/vec/exec/join/process_hash_table_probe_impl.h
@@ -213,6 +213,7 @@ Status ProcessHashTableProbe::do_process(HashTableType& hash
}
auto& mcol = mutable_block.mutable_columns();
+const bool has_mark_join_conjunct = !_parent->_mark_join_conjuncts.empty();
int current_offset = 0;
if constexpr ((JoinOpType == doris::TJoinOp::NULL_AWARE_LEFT_ANTI_JOIN ||
@@ -258,7 +259,7 @@ Status ProcessHashTableProbe::do_process(HashTableType& hash
need_null_map_for_probe &&
ignore_null > (hash_table_ctx.keys,
hash_table_ctx.bucket_nums.data(),
probe_index, build_index, probe_rows,
_probe_indexs.data(),
- _probe_visited, _build_indexs.data());
+ _probe_visited, _build_inde
(doris) 05/12: [case](regression) Add hdfs backup restore case (#34716)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e2614d453a802eafa953748468dcb9ca291e8a96
Author: walter
AuthorDate: Sat May 11 19:54:59 2024 +0800
[case](regression) Add hdfs backup restore case (#34716)
---
.../backup_restore/test_backup_restore_hdfs.groovy | 107 +
1 file changed, 107 insertions(+)
diff --git
a/regression-test/suites/backup_restore/test_backup_restore_hdfs.groovy
b/regression-test/suites/backup_restore/test_backup_restore_hdfs.groovy
new file mode 100644
index 000..239fbde6b53
--- /dev/null
+++ b/regression-test/suites/backup_restore/test_backup_restore_hdfs.groovy
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_backup_restore_hdfs", "backup_restore") {
+String suiteName = "test_backup_restore_hdfs"
+String repoName = "${suiteName}_repo"
+String dbName = "${suiteName}_db"
+String tableName = "${suiteName}_table"
+String snapshotName = "${suiteName}_snapshot_" +
UUID.randomUUID().toString().replace('-', '')
+
+if (!enableHdfs()) {
+logger.info("hdfs is not enabled, skip this case")
+return
+}
+
+def syncer = getSyncer()
+syncer.createHdfsRepository(repoName)
+
+// create repository for fs broker:
+//
+// String hdfsFs = getHdfsFs()
+// String hdfsUser = getHdfsUser()
+// String dataDir = getHdfsDataDir()
+// String prefix = "test"
+
+// try_sql "DROP REPOSITORY `${repoName}`"
+// sql """
+// CREATE REPOSITORY `${repoName}`
+// WITH broker `broker_0`
+// ON LOCATION "${dataDir}/${prefix}/${repoName}"
+// PROPERTIES
+// (
+// "fs.defaultFS" = "${hdfsFs}",
+// "hadoop.username" = "${hdfsUser}"
+// )
+// """
+
+sql "CREATE DATABASE IF NOT EXISTS ${dbName}"
+sql "DROP TABLE IF EXISTS ${dbName}.${tableName}"
+sql """
+CREATE TABLE ${dbName}.${tableName} (
+`id` LARGEINT NOT NULL,
+`count` LARGEINT SUM DEFAULT "0")
+AGGREGATE KEY(`id`)
+DISTRIBUTED BY HASH(`id`) BUCKETS 2
+PROPERTIES
+(
+"replication_num" = "1"
+)
+"""
+
+List values = []
+for (int i = 1; i <= 10; ++i) {
+values.add("(${i}, ${i})")
+}
+sql "INSERT INTO ${dbName}.${tableName} VALUES ${values.join(",")}"
+def result = sql "SELECT * FROM ${dbName}.${tableName}"
+assertEquals(result.size(), values.size());
+
+sql """
+BACKUP SNAPSHOT ${dbName}.${snapshotName}
+TO `${repoName}`
+ON (${tableName})
+"""
+
+syncer.waitSnapshotFinish(dbName)
+
+def snapshot = syncer.getSnapshotTimestamp(repoName, snapshotName)
+assertTrue(snapshot != null)
+
+sql "TRUNCATE TABLE ${dbName}.${tableName}"
+
+sql """
+RESTORE SNAPSHOT ${dbName}.${snapshotName}
+FROM `${repoName}`
+ON ( `${tableName}`)
+PROPERTIES
+(
+"backup_timestamp" = "${snapshot}",
+"reserve_replica" = "true"
+)
+"""
+
+syncer.waitAllRestoreFinish(dbName)
+
+result = sql "SELECT * FROM ${dbName}.${tableName}"
+assertEquals(result.size(), values.size());
+
+sql "DROP TABLE ${dbName}.${tableName} FORCE"
+sql "DROP DATABASE ${dbName} FORCE"
+sql "DROP REPOSITORY `${repoName}`"
+}
+
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 10/12: [WIP](test) remove enable_nereids_planner in regression cases (part 4) (#34642)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 1545d96617c6bf5cac7605cbc9884f777e5ab650
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Mon May 13 15:12:12 2024 +0800
[WIP](test) remove enable_nereids_planner in regression cases (part 4)
(#34642)
before PR are
#34417
#34490
#34558
---
.../correctness_p0/test_group_having_alias.out | 2 -
.../test_array_functions_array_intersect_sort.out | 110 ++---
.../data/javaudf_p0/test_javaudf_with_decimal.out | 8 +-
.../suites/correctness_p0/test_cast_decimal.groovy | 14 ---
.../correctness_p0/test_group_having_alias.groovy | 20
...est_array_functions_array_intersect_sort.groovy | 97 +++---
...l_tvf_with_complex_type_insertinto_doris.groovy | 1 -
.../tvf/test_tvf_view_count_p2.groovy | 16 +--
.../javaudf_p0/test_javaudf_with_decimal.groovy| 37 +--
...st_array_with_single_row_max_than_uint16.groovy | 1 -
10 files changed, 32 insertions(+), 274 deletions(-)
diff --git a/regression-test/data/correctness_p0/test_group_having_alias.out
b/regression-test/data/correctness_p0/test_group_having_alias.out
index 727040875a7..0b711d92ad6 100644
--- a/regression-test/data/correctness_p0/test_group_having_alias.out
+++ b/regression-test/data/correctness_p0/test_group_having_alias.out
@@ -10,8 +10,6 @@
202245
202245
--- !sql --
-
-- !sql --
202245 3
diff --git
a/regression-test/data/datatype_p0/nested_types/query/array_functions/test_array_functions_array_intersect_sort.out
b/regression-test/data/datatype_p0/nested_types/query/array_functions/test_array_functions_array_intersect_sort.out
index 0127b7b25db..0ede6c3b7a2 100644
---
a/regression-test/data/datatype_p0/nested_types/query/array_functions/test_array_functions_array_intersect_sort.out
+++
b/regression-test/data/datatype_p0/nested_types/query/array_functions/test_array_functions_array_intersect_sort.out
@@ -1,109 +1,43 @@
-- This file is automatically generated. You should know what you did if you
want to edit this
--- !oLd_sql --
+-- !nereid_sql --
array_intersect-array-sort
--- !old_sql --
-[1, 2]
-[1, 2]
-[1]
-[]
-
--- !old_sql --
-[]
-[]
-[]
-[]
-
--- !old_sql --
-[]
-[]
-[]
-[]
-
--- !old_sql --
-[1, 2]
-[1, 2]
-[1]
-[]
-
--- !old_sql --
-[1, 2]
-[1, 2]
-[1, 2]
-[1, 2]
-
--- !old_sql --
-[]
-[]
-[]
-[]
-
--- !old_sql --
+-- !nereid_sql --
[1, 2]
[1, 2]
[1]
[]
--- !old_sql --
+-- !nereid_sql --
[]
[]
[]
[]
--- !old_sql --
+-- !nereid_sql --
[]
[]
[]
[]
--- !old_sql --
+-- !nereid_sql --
[1, 2]
[1, 2]
[1]
[]
--- !old_sql --
+-- !nereid_sql --
[1, 2]
[1, 2]
[1, 2]
[1, 2]
--- !old_sql --
-[]
-[]
-[]
-[]
-
--- !old_sql --
-[]
-
--- !old_sql --
-[]
-
--- !old_sql --
-["a"]
-
--- !old_sql --
+-- !nereid_sql --
[]
-
--- !old_sql --
-[1]
-
--- !old_sql --
[]
-
--- !old_sql --
-[2]
-
--- !old_sql --
[]
-
--- !old_sql --
[]
--- !nereid_sql --
-array_intersect-array-sort
-
-- !nereid_sql --
[1, 2]
[1, 2]
@@ -141,53 +75,29 @@ array_intersect-array-sort
[]
-- !nereid_sql --
-[1, 2]
-[1, 2]
-[1]
[]
-- !nereid_sql --
[]
-[]
-[]
-[]
-- !nereid_sql --
-[]
-[]
-[]
-[]
+["a"]
-- !nereid_sql --
-[1, 2]
-[1, 2]
-[1]
[]
-- !nereid_sql --
-[1, 2]
-[1, 2]
-[1, 2]
-[1, 2]
+[1]
-- !nereid_sql --
[]
-[]
-[]
-[]
-- !nereid_sql --
-[]
+[2]
-- !nereid_sql --
[]
--- !nereid_sql --
-["a"]
-
-- !nereid_sql --
[]
--- !nereid_sql --
-[1]
-
diff --git a/regression-test/data/javaudf_p0/test_javaudf_with_decimal.out
b/regression-test/data/javaudf_p0/test_javaudf_with_decimal.out
index 4a05a63689f..abcbe67e12c 100644
--- a/regression-test/data/javaudf_p0/test_javaudf_with_decimal.out
+++ b/regression-test/data/javaudf_p0/test_javaudf_with_decimal.out
@@ -1,13 +1,9 @@
-- This file is automatically generated. You should know what you did if you
want to edit this
--- !select_1 --
+-- !getarrscale --
[1.123, 1.123] 3
[2.123, 2.123] 3
--- !select_2 --
-{1.123:1.123457} 9
-{2.123:2.123457} 9
-
--- !select_3 --
+-- !retscale --
1 {1.00:1.111000}
2 {2.00:2.222000}
diff --git a/regression-test/suites/correctness_p0/test_cast_decimal.groovy
b/regression-test/suites/correctness_p0/test_cast_decimal.groovy
index 21a1ab6d0c3..ec9a8434c46 100644
--- a/regression-test/suites/correctness_p0/test_cast_decimal.groovy
+++ b/regression-test/suites/correctness_p0/test_cast_decimal.groovy
@@ -16,20 +16,6 @@
// under the License.
suite("test_cast_decimal") {
-sql """
-set enable_nereids_planner=true;
-"""
-
-explain {
-sql """select cast(32123.34212456734 as decimal(3,2));"""
-contains "cast(32123.34212456734 as DECIMALV3(3, 2))"
-}
-
-
-sql """
-set enable_nereids_planner=false;
-"""
-
explain
(doris) 04/12: [fix](test) fix s3 load test failed (#34671)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 6f91e9cc4dcb670f34fa381ac91a7c3dcf424c6c
Author: HHoflittlefish777 <77738092+hhoflittlefish...@users.noreply.github.com>
AuthorDate: Sat May 11 17:23:19 2024 +0800
[fix](test) fix s3 load test failed (#34671)
---
.../load_p0/broker_load/test_s3_load_with_load_parallelism.groovy | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git
a/regression-test/suites/load_p0/broker_load/test_s3_load_with_load_parallelism.groovy
b/regression-test/suites/load_p0/broker_load/test_s3_load_with_load_parallelism.groovy
index be0d7f9c348..b1d1782b501 100644
---
a/regression-test/suites/load_p0/broker_load/test_s3_load_with_load_parallelism.groovy
+++
b/regression-test/suites/load_p0/broker_load/test_s3_load_with_load_parallelism.groovy
@@ -45,8 +45,10 @@ suite("test_s3_load_with_load_parallelism", "load_p0") {
// attributesList.add(new
LoadAttributes("s3://doris-build-1308700295/regression/load/data/enclose_not_trim_quotes.csv",
// "${tableName}", "", "COLUMNS TERMINATED BY \",\"", "FORMAT AS
\"CSV\"", "(k1,k2,v1,v2,v3,v4)",
// "PROPERTIES (\"enclose\" = \"\\\"\", \"escape\" =
\"\")").addProperties("trim_double_quotes", "false"))
+
+def bucket = getS3BucketName()
-attributesList.add(new
LoadAttributes("s3://test-for-student-1308700295/regression/segcompaction/segcompaction.orc",
+attributesList.add(new
LoadAttributes("s3://${bucket}/regression/segcompaction/segcompaction.orc",
"${tableName}", "", "", "FORMAT AS \"ORC\"", "(col_0, col_1, col_2,
col_3, col_4, col_5, col_6, col_7, col_8, col_9, col_10, col_11, col_12,
col_13, col_14, col_15, col_16, col_17, col_18, col_19, col_20, col_21, col_22,
col_23, col_24, col_25, col_26, col_27, col_28, col_29, col_30, col_31, col_32,
col_33, col_34, col_35, col_36, col_37, col_38, col_39, col_40, col_41, col_42,
col_43, col_44, col_45, col_46, col_47, col_48, col_49)",
"").addProperties("load_parallelism", "3"))
def ak = getS3AK()
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 08/12: [fix](Nereids) cast from json should always nullable (#34707)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c71d0b6b223d092d98bc42b3b81f3d5636e9e63e
Author: morrySnow <101034200+morrys...@users.noreply.github.com>
AuthorDate: Mon May 13 14:42:04 2024 +0800
[fix](Nereids) cast from json should always nullable (#34707)
---
.../apache/doris/nereids/trees/expressions/Cast.java | 2 ++
regression-test/suites/query_p0/cast/test_cast.groovy | 18 ++
2 files changed, 20 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Cast.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Cast.java
index 76cb1826e55..62bd3639b5a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Cast.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/Cast.java
@@ -79,6 +79,8 @@ public class Cast extends Expression implements
UnaryExpression {
return true;
} else if (!childDataType.isTimeLikeType() &&
targetType.isTimeLikeType()) {
return true;
+} else if (childDataType.isJsonType()) {
+return true;
} else {
return child().nullable();
}
diff --git a/regression-test/suites/query_p0/cast/test_cast.groovy
b/regression-test/suites/query_p0/cast/test_cast.groovy
index c0df52b39e7..2fe4d53eb80 100644
--- a/regression-test/suites/query_p0/cast/test_cast.groovy
+++ b/regression-test/suites/query_p0/cast/test_cast.groovy
@@ -145,4 +145,22 @@ suite('test_cast', "arrow_flight_sql") {
sql "select * from ${tbl} where case when k0 = 101 then 'true' else 1
end"
result([[101]])
}
+
+sql "DROP TABLE IF EXISTS test_json"
+sql """
+CREATE TABLE IF NOT EXISTS test_json (
+ id INT not null,
+ j JSON not null
+)
+DUPLICATE KEY(id)
+DISTRIBUTED BY HASH(id) BUCKETS 10
+PROPERTIES("replication_num" = "1");
+"""
+
+sql """
+INSERT INTO test_json VALUES(26, '{"k1":"v1", "k2": 200}');
+"""
+sql "sync"
+sql "Select cast(j as int) from test_json"
+sql "DROP TABLE IF EXISTS test_json"
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 01/12: [Performance](Variant) Improve load performance for variant type (#33890)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 691f3c5ee7a36a4ae374dbef731f25ce5c2bf20d
Author: lihangyu <15605149...@163.com>
AuthorDate: Sat May 11 11:31:24 2024 +0800
[Performance](Variant) Improve load performance for variant type (#33890)
1. remove phmap for padding rows
2. add SimpleFieldVisitorToScarlarType for short circuit type deducing
3. correct type coercion for conflict types bettween integers
4. improve nullable column performance
5. remove shared_ptr dependancy for DataType use TypeIndex instead
6. Optimization by caching the order of fields (which is almost always the
same)
and a quick check to match the next expected field, instead of searching
the hash table.
benchmark:
In clickbench data, load performance:
12m36.799s ->7m10.934s about 43% latency reduce
In variant_p2/performance.groovy:
3min44s20 -> 1min15s80 about 66% latency reducy
---
be/src/vec/columns/column_object.cpp | 189 -
be/src/vec/columns/column_object.h | 28 ++-
be/src/vec/common/schema_util.cpp | 57 ---
be/src/vec/common/schema_util.h| 4 +-
be/src/vec/core/field.h| 5 +
be/src/vec/json/parse2column.cpp | 22 +--
.../suites/variant_p2/performance.groovy | 36
7 files changed, 248 insertions(+), 93 deletions(-)
diff --git a/be/src/vec/columns/column_object.cpp
b/be/src/vec/columns/column_object.cpp
index ddb5bee6e01..3bae978f4d3 100644
--- a/be/src/vec/columns/column_object.cpp
+++ b/be/src/vec/columns/column_object.cpp
@@ -45,6 +45,7 @@
#include "util/defer_op.h"
#include "util/simd/bits.h"
#include "vec/aggregate_functions/aggregate_function.h"
+#include "vec/aggregate_functions/helpers.h"
#include "vec/columns/column.h"
#include "vec/columns/column_array.h"
#include "vec/columns/column_nullable.h"
@@ -56,6 +57,7 @@
#include "vec/common/field_visitors.h"
#include "vec/common/schema_util.h"
#include "vec/common/string_buffer.hpp"
+#include "vec/common/string_ref.h"
#include "vec/core/column_with_type_and_name.h"
#include "vec/core/field.h"
#include "vec/core/types.h"
@@ -68,6 +70,7 @@
#include "vec/data_types/data_type_nothing.h"
#include "vec/data_types/data_type_nullable.h"
#include "vec/data_types/get_least_supertype.h"
+#include "vec/json/path_in_data.h"
#ifdef __AVX2__
#include "util/jsonb_parser_simd.h"
@@ -78,23 +81,22 @@
namespace doris::vectorized {
namespace {
-DataTypePtr create_array_of_type(DataTypePtr type, size_t num_dimensions, bool
is_nullable) {
-const DataTypeNullable* nullable = typeid_cast(type.get());
-if ((nullable &&
- typeid_cast(nullable->get_nested_type().get())) ||
-typeid_cast(type.get())) {
+DataTypePtr create_array_of_type(TypeIndex type, size_t num_dimensions, bool
is_nullable) {
+if (type == ColumnObject::MOST_COMMON_TYPE_ID) {
// JSONB type MUST NOT wrapped in ARRAY column, it should be top level.
// So we ignored num_dimensions.
-return type;
+return is_nullable ?
make_nullable(std::make_shared())
+ : std::make_shared();
}
+DataTypePtr result = DataTypeFactory::instance().create_data_type(type,
is_nullable);
for (size_t i = 0; i < num_dimensions; ++i) {
-type = std::make_shared(std::move(type));
+result = std::make_shared(result);
if (is_nullable) {
// wrap array with nullable
-type = make_nullable(type);
+result = make_nullable(result);
}
}
-return type;
+return result;
}
DataTypePtr get_base_type_of_array(const DataTypePtr& type) {
@@ -149,6 +151,63 @@ public:
}
};
+// Visitor that allows to get type of scalar field
+// but exclude fields contain complex field.This is a faster version
+// for FieldVisitorToScalarType which does not support complex field.
+class SimpleFieldVisitorToScalarType : public StaticVisitor {
+public:
+size_t operator()(const Array& x) {
+throw doris::Exception(ErrorCode::INVALID_ARGUMENT, "Array type is not
supported");
+}
+size_t operator()(const UInt64& x) {
+if (x <= std::numeric_limits::max()) {
+type = TypeIndex::Int8;
+} else if (x <= std::numeric_limits::max()) {
+type = TypeIndex::Int16;
+} else if (x <= std::numeric_limits::max()) {
+type = TypeIndex::Int32;
+} else {
+type = TypeIndex::Int64;
+}
+return 1;
+}
+size_t operator()(const Int64& x) {
+if (x <= std::numeric_limits::max() && x >=
std::numeric_limits::min()) {
+type = TypeIndex::Int8;
+} else if (x <= std::numeric_limits::max() &&
+ x >= st
(doris) 11/12: [improvement](inverted index) Disable the use of skipping write index on load (#34719)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit b6409f5584107db86ad9c3b09dde556663c6532a
Author: qiye
AuthorDate: Mon May 13 15:27:51 2024 +0800
[improvement](inverted index) Disable the use of skipping write index on
load (#34719)
When `skip_write_index_on_load` is turned on, users will get an error when
querying for the latest data(not compacted), giving them a bad experience. And
we can use `inverted_index_ram_dir_enable = true` and
`inverted_index_storage_format=V2` to reduce IO and CPU consumption. So we
disable it now.
1. Disable setting `skip_write_index_on_load` to `true` in create table
stmt.
2. Disable setting `skip_write_index_on_load` to `true` in alter table
properties stmt. You can still alter `skip_write_index_on_load` to `false`.
Co-authored-by: Luennng
---
.../java/org/apache/doris/analysis/ModifyTablePropertiesClause.java | 5 +
.../src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java | 3 ++-
.../src/main/java/org/apache/doris/datasource/InternalCatalog.java | 3 +--
regression-test/suites/datatype_p0/scalar_types/load.groovy | 2 +-
4 files changed, 9 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java
index c2bc7bc7d0d..e0aa994c6d3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/analysis/ModifyTablePropertiesClause.java
@@ -240,6 +240,11 @@ public class ModifyTablePropertiesClause extends
AlterTableClause {
this.needTableStable = false;
this.opType = AlterOpType.MODIFY_TABLE_PROPERTY_SYNC;
} else if
(properties.containsKey(PropertyAnalyzer.PROPERTIES_SKIP_WRITE_INDEX_ON_LOAD)) {
+if
(properties.get(PropertyAnalyzer.PROPERTIES_SKIP_WRITE_INDEX_ON_LOAD).equalsIgnoreCase("true"))
{
+throw new AnalysisException(
+"Property "
++ PropertyAnalyzer.PROPERTIES_SKIP_WRITE_INDEX_ON_LOAD + "
is forbidden now");
+}
if
(!properties.get(PropertyAnalyzer.PROPERTIES_SKIP_WRITE_INDEX_ON_LOAD).equalsIgnoreCase("true")
&& !properties.get(PropertyAnalyzer
.PROPERTIES_SKIP_WRITE_INDEX_ON_LOAD).equalsIgnoreCase("false")) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
b/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
index ba4434349cb..07236da2ee3 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java
@@ -705,7 +705,8 @@ public class PropertyAnalyzer {
}
properties.remove(PROPERTIES_SKIP_WRITE_INDEX_ON_LOAD);
if (value.equalsIgnoreCase("true")) {
-return true;
+throw new AnalysisException("Property " +
PROPERTIES_SKIP_WRITE_INDEX_ON_LOAD
++ " is forbidden now.");
} else if (value.equalsIgnoreCase("false")) {
return false;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
index ee3b6ee9651..41ef985f17f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
@@ -2405,8 +2405,7 @@ public class InternalCatalog implements
CatalogIf {
olapTable.setStoreRowColumn(storeRowColumn);
// set skip inverted index on load
-boolean skipWriteIndexOnLoad =
PropertyAnalyzer.analyzeBooleanProp(properties,
-PropertyAnalyzer.PROPERTIES_SKIP_WRITE_INDEX_ON_LOAD, false);
+boolean skipWriteIndexOnLoad =
PropertyAnalyzer.analyzeSkipWriteIndexOnLoad(properties);
olapTable.setSkipWriteIndexOnLoad(skipWriteIndexOnLoad);
boolean isMutable = PropertyAnalyzer.analyzeBooleanProp(properties,
PropertyAnalyzer.PROPERTIES_MUTABLE, true);
diff --git a/regression-test/suites/datatype_p0/scalar_types/load.groovy
b/regression-test/suites/datatype_p0/scalar_types/load.groovy
index 91cd4eb1f1f..4c7c7ee9366 100644
--- a/regression-test/suites/datatype_p0/scalar_types/load.groovy
+++ b/regression-test/suites/datatype_p0/scalar_types/load.groovy
@@ -448,7 +448,7 @@ suite("test_scalar_types_load", "p0") {
DUPLICATE KEY(`k1`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`k1`) BUCKETS 10
-PROPERTIES("replication_num" = "1", "skip_write_index_on_load" =
"true");
+PROPERTI
(doris) 03/12: [fix](function) json_object can not input null value (#34591)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 876248aa4e382d6268b44811909e6fecc8002ed0
Author: Mryange <59914473+mrya...@users.noreply.github.com>
AuthorDate: Sat May 11 14:17:43 2024 +0800
[fix](function) json_object can not input null value (#34591)
---
be/src/vec/functions/function_json.cpp | 24 +-
.../json_function/test_query_json_object.groovy| 4
2 files changed, 27 insertions(+), 1 deletion(-)
diff --git a/be/src/vec/functions/function_json.cpp
b/be/src/vec/functions/function_json.cpp
index 4418561d881..e7c2fc1781d 100644
--- a/be/src/vec/functions/function_json.cpp
+++ b/be/src/vec/functions/function_json.cpp
@@ -619,6 +619,7 @@ struct ExecuteReducer {
struct FunctionJsonArrayImpl {
static constexpr auto name = "json_array";
+static constexpr auto must_not_null = false;
template
using Reducer = ExecuteReducer;
@@ -654,7 +655,7 @@ struct FunctionJsonArrayImpl {
struct FunctionJsonObjectImpl {
static constexpr auto name = "json_object";
-
+static constexpr auto must_not_null = true;
template
using Reducer = ExecuteReducer;
@@ -743,6 +744,9 @@ public:
data_columns.push_back(assert_cast(column_ptrs.back().get()));
}
}
+if (SpecificImpl::must_not_null) {
+RETURN_IF_ERROR(check_keys_all_not_null(nullmaps,
input_rows_count, arguments.size()));
+}
execute(data_columns,
*assert_cast(result_column.get()), input_rows_count,
nullmaps);
block.get_by_position(result).column = std::move(result_column);
@@ -774,6 +778,24 @@ public:
result_column.insert_data(buf.GetString(), buf.GetSize());
}
}
+
+static Status check_keys_all_not_null(const std::vector& nullmaps, int size,
+ size_t args) {
+for (int i = 0; i < args; i += 2) {
+const auto* null_map = nullmaps[i];
+if (null_map) {
+const bool not_null_num =
+
simd::count_zero_num((int8_t*)null_map->get_data().data(), size);
+if (not_null_num < size) {
+return Status::InternalError(
+"function {} can not input null value , JSON
documents may not contain "
+"NULL member names.",
+name);
+}
+}
+}
+return Status::OK();
+}
};
struct FunctionJsonQuoteImpl {
diff --git
a/regression-test/suites/query_p0/sql_functions/json_function/test_query_json_object.groovy
b/regression-test/suites/query_p0/sql_functions/json_function/test_query_json_object.groovy
index 2a12d69ffe0..e4ebe839276 100644
---
a/regression-test/suites/query_p0/sql_functions/json_function/test_query_json_object.groovy
+++
b/regression-test/suites/query_p0/sql_functions/json_function/test_query_json_object.groovy
@@ -41,5 +41,9 @@ suite("test_query_json_object", "query") {
sql "insert into ${tableName} values(4,null,null,'test','2022-01-01
11:11:11');"
sql "insert into ${tableName} values(5,1,true,'test','2022-01-01
11:11:11');"
qt_sql1 "select json_object('k0',k0,'k1',k1,'k2',k2,'k3',k3,'k4',k4,'k5',
null,'k6','k6') from ${tableName} order by k0;"
+test {
+sql """select k0,json_object(k3,123) from ${tableName} order by k0;"""
+exception "[CANCELLED][INTERNAL_ERROR] function json_object can not
input null value , JSON documents may not contain NULL member names."
+}
sql "DROP TABLE ${tableName};"
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 02/12: Fix failed p2 hive statistics case. (#34663)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 7e967e53b83d451dff4dd14da4162a53aba398da
Author: Jibing-Li <64681310+jibing...@users.noreply.github.com>
AuthorDate: Sat May 11 14:05:22 2024 +0800
Fix failed p2 hive statistics case. (#34663)
---
.../hive/test_hive_partition_column_analyze.groovy | 390 +++--
.../hive/test_hive_partition_statistic.groovy | 3 +
2 files changed, 200 insertions(+), 193 deletions(-)
diff --git
a/regression-test/suites/external_table_p2/hive/test_hive_partition_column_analyze.groovy
b/regression-test/suites/external_table_p2/hive/test_hive_partition_column_analyze.groovy
index 336e2e8b784..d4b1fa3eca4 100644
---
a/regression-test/suites/external_table_p2/hive/test_hive_partition_column_analyze.groovy
+++
b/regression-test/suites/external_table_p2/hive/test_hive_partition_column_analyze.groovy
@@ -31,199 +31,203 @@ suite("test_hive_partition_column_analyze",
"p2,external,hive,external_remote,ex
"""
logger.info("catalog " + catalog_name + " created")
-// Test analyze table without init.
-sql """analyze table
${catalog_name}.multi_partition.multi_partition_parquet (event_day) with sync"""
-sql """analyze table
${catalog_name}.multi_partition.multi_partition_orc (event_day) with sync"""
-
-sql """switch ${catalog_name};"""
-logger.info("switched to catalog " + catalog_name)
-sql """use multi_partition;"""
-def result = sql """show column stats multi_partition_parquet
(event_day)"""
-assertEquals(result.size(), 1)
-assertEquals(result[0][0], "event_day")
-assertEquals(result[0][2], "3.83714205E8")
-assertEquals(result[0][3], "99949.0")
-assertEquals(result[0][4], "0.0")
-assertEquals(result[0][5], "3.83714205E9")
-assertEquals(result[0][6], "10.0")
-assertEquals(result[0][7], "\'1749-09-24\'")
-assertEquals(result[0][8], "\'2023-05-26\'")
-
-result = sql """show column stats multi_partition_orc (event_day)"""
-assertEquals(result.size(), 1)
-assertEquals(result[0][0], "event_day")
-assertEquals(result[0][2], "1.9007155E8")
-assertEquals(result[0][3], "99949.0")
-assertEquals(result[0][4], "0.0")
-assertEquals(result[0][5], "1.9007155E9")
-assertEquals(result[0][6], "10.0")
-assertEquals(result[0][7], "\'1749-09-24\'")
-assertEquals(result[0][8], "\'2023-05-26\'")
-
-sql """analyze table ${catalog_name}.partition_type.tinyint_partition
(tinyint_part) with sync"""
-sql """analyze table ${catalog_name}.partition_type.smallint_partition
(smallint_part) with sync"""
-sql """analyze table ${catalog_name}.partition_type.int_partition
(int_part) with sync"""
-sql """analyze table ${catalog_name}.partition_type.bigint_partition
(bigint_part) with sync"""
-sql """analyze table ${catalog_name}.partition_type.char_partition
(char_part) with sync"""
-sql """analyze table ${catalog_name}.partition_type.varchar_partition
(varchar_part) with sync"""
-sql """analyze table ${catalog_name}.partition_type.string_partition
(string_part) with sync"""
-sql """analyze table ${catalog_name}.partition_type.date_partition
(date_part) with sync"""
-sql """analyze table ${catalog_name}.partition_type.float_partition
(float_part) with sync"""
-sql """analyze table ${catalog_name}.partition_type.double_partition
(double_part) with sync"""
-sql """analyze table ${catalog_name}.partition_type.decimal_partition
(decimal_part) with sync"""
-sql """analyze table ${catalog_name}.partition_type.two_partition
(part1, part2) with sync"""
-
-sql """use partition_type;"""
-
-result = sql """show column stats tinyint_partition (tinyint_part)"""
-assertEquals(result.size(), 1)
-assertEquals(result[0][0], "tinyint_part")
-assertEquals(result[0][2], "141474.0")
-assertEquals(result[0][3], "100.0")
-assertEquals(result[0][4], "0.0")
-assertEquals(result[0][5], "141474.0")
-assertEquals(result[0][6], "1.0")
-assertEquals(result[0][7], "1")
-assertEquals(result[0][8], "100")
-
-result = sql """show column stats smallint_partition (smallint_part)"""
-assertEquals(result.size(), 1)
-assertEquals(result[0][0], "smallint_part")
-assertEquals(result[0][2], "141474.0")
-assertEquals(result[0][3], "100.0")
-assertEquals(result[0][4], "0.0")
-assertEquals(result[0][5], "282948.0")
-assertEquals(result[0][6], "2.0")
-assertEquals(result[0][7], "1")
-assertEquals(result[0][8], "100")
-
-result = sql """show column stats int_partition (int_part)"""
-assertEquals
(doris) 09/12: [test](case) error format case in test_query_json_object (#34722)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 46bf43130f78de6cbdce2bbd597bc40822e94b8e
Author: Mryange <59914473+mrya...@users.noreply.github.com>
AuthorDate: Mon May 13 15:05:53 2024 +0800
[test](case) error format case in test_query_json_object (#34722)
error format case in test_query_json_object
---
.../query_p0/sql_functions/json_function/test_query_json_object.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/regression-test/suites/query_p0/sql_functions/json_function/test_query_json_object.groovy
b/regression-test/suites/query_p0/sql_functions/json_function/test_query_json_object.groovy
index e4ebe839276..c3b1e35b5e6 100644
---
a/regression-test/suites/query_p0/sql_functions/json_function/test_query_json_object.groovy
+++
b/regression-test/suites/query_p0/sql_functions/json_function/test_query_json_object.groovy
@@ -43,7 +43,7 @@ suite("test_query_json_object", "query") {
qt_sql1 "select json_object('k0',k0,'k1',k1,'k2',k2,'k3',k3,'k4',k4,'k5',
null,'k6','k6') from ${tableName} order by k0;"
test {
sql """select k0,json_object(k3,123) from ${tableName} order by k0;"""
-exception "[CANCELLED][INTERNAL_ERROR] function json_object can not
input null value , JSON documents may not contain NULL member names."
+exception "function json_object can not input null value , JSON
documents may not contain NULL member names."
}
sql "DROP TABLE ${tableName};"
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature][external catalog/lakesoul] support lakesoul catalog [doris]
morningman commented on PR #32164: URL: https://github.com/apache/doris/pull/32164#issuecomment-2118760982 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
[PR] [doc](udf) fix doc indent [doris-website]
morningman opened a new pull request, #667: URL: https://github.com/apache/doris-website/pull/667 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](hive)Misspelling of class names [doris]
morningman commented on PR #34981: URL: https://github.com/apache/doris/pull/34981#issuecomment-2118765111 I will merge it first quickly and add regression test later -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](hive)Misspelling of class names [doris]
morningman commented on PR #34981: URL: https://github.com/apache/doris/pull/34981#issuecomment-2118765198 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](hive)Misspelling of class names [doris]
github-actions[bot] commented on PR #34981: URL: https://github.com/apache/doris/pull/34981#issuecomment-2118765344 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [bugfix](hive)Misspelling of class names [doris]
github-actions[bot] commented on PR #34981: URL: https://github.com/apache/doris/pull/34981#issuecomment-2118765356 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](agg) memory leak issue in agg operator [doris]
doris-robot commented on PR #35037: URL: https://github.com/apache/doris/pull/35037#issuecomment-2118765449 TeamCity be ut coverage result: Function Coverage: 35.71% (9008/25228) Line Coverage: 27.36% (74494/272257) Region Coverage: 26.61% (38530/144799) Branch Coverage: 23.43% (19657/83888) Coverage Report: http://coverage.selectdb-in.cc/coverage/98ba12d5d24e9467769f445b681d3a6c21d90769_98ba12d5d24e9467769f445b681d3a6c21d90769/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (80dd027ce2e -> db273d578f0)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from 80dd027ce2e [opt](join) For left semi/anti join without mark join conjunct and without other conjucnts, stop probing after matching one row (#34703) new 6c515e0c76a [fix](group commit) Make compatibility issues on serializing and deserializing wal file more clear (#34793) new f7801948ad8 fix backup and restore failed between force_replication_allocation setted and not setted clusters (#34608) new 30a036e7a48 [feature](mtmv) create mtmv support partitions rollup (#31812) new 38bac76b376 [opt](mtmv) Cache materialization check result for performance (#34301) new 6b1c441258a [fix](group_commit) Wal reader should check block length to avoid reading empty block (#34792) new 1e53a2a81d1 [Improve](inverted index) improve query performance by not using output index result column (#34281) new 4a8df535537 [Chore](rollup) check duplicate column name when create table with rollup (#34827) new 71caf88ec15 [opt](mtmv) Optimize the logic of slot mapping generate for performance (#34597) new 5719f6ff0c6 [fix](planner) fix date_xxx functions without complete function signature (#34761) new 8264078a9a8 [fix](nereids)4 phase agg may lost parameter in some case (#34816) new a07876e807d [fix](planner)correlated predicate should include isnull predicate (#34833) new 849eeb39e9c [fix](load) skip sending cancel rpc if VNodeChannel is not inited (#34897) new 4b96f9834fd [fix](move-memtable) change brpc connection type to single (#34883) new dff61715461 [fix](auto inc) db_id and table_id should be int64_t instead of int32_t (#34912) new 0febfc10e45 [Fix](inverted index) fix wrong fs in inverted_index_file_writer (#34903) new 5d1f5968ebe [fix](case)fix PolicyTest testMergeFilterNereidsPlanner failed (#34637) new b51a4212d68 [fix](txn insert) Fix txn insert values error when connect to follower fe (#34950) new db273d578f0 [Fix](tablet id) use int64_t instead of int32_t or uint32_t for tablet_id (#34962) The 18 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: be/src/agent/be_exec_version_manager.h | 1 + be/src/olap/iterators.h| 2 +- be/src/olap/rowset/segment_v2/column_reader.h | 4 +- .../segment_v2/inverted_index_file_writer.cpp | 2 +- be/src/olap/rowset/segment_v2/segment_iterator.cpp | 4 +- be/src/olap/rowset/segment_v2/segment_iterator.h | 4 +- be/src/olap/schema_cache.cpp | 4 +- be/src/olap/schema_cache.h | 4 +- be/src/olap/tablet_schema.h| 12 +- be/src/olap/utils.h| 4 +- be/src/olap/wal/wal_reader.cpp | 3 + be/src/runtime/descriptors.h | 4 +- be/src/util/brpc_client_cache.h| 6 +- be/src/vec/exec/format/wal/wal_reader.cpp | 11 +- be/src/vec/sink/autoinc_buffer.cpp | 7 + be/src/vec/sink/load_stream_stub.cpp | 6 +- be/src/vec/sink/writer/vtablet_writer.cpp | 4 + .../antlr4/org/apache/doris/nereids/DorisParser.g4 | 7 +- .../org/apache/doris/analysis/RestoreStmt.java | 5 + .../org/apache/doris/analysis/StmtRewriter.java| 3 +- .../org/apache/doris/backup/BackupJobInfo.java | 4 + .../java/org/apache/doris/backup/RestoreJob.java | 4 + .../main/java/org/apache/doris/catalog/MTMV.java | 40 +-- .../apache/doris/job/extensions/mtmv/MTMVTask.java | 6 +- .../doris/mtmv/MTMVPartitionExprDateTrunc.java | 228 .../doris/mtmv/MTMVPartitionExprFactory.java | 39 +++ .../doris/mtmv/MTMVPartitionExprService.java | 60 + .../org/apache/doris/mtmv/MTMVPartitionInfo.java | 39 ++- .../org/apache/doris/mtmv/MTMVPartitionUtil.java | 79 +++--- .../MTMVRelatedPartitionDescGeneratorService.java | 38 +++ .../MTMVRelatedPartitionDescInitGenerator.java | 34 +++ ...latedPartitionDescOnePartitionColGenerator.java | 67 + .../MTMVRelatedPartitionDescRollUpGenerator.java | 142 ++ ...TMVRelatedPartitionDescSyncLimitGenerator.java} | 145 +++--- .../org/apache/doris/mtmv/MTMVRelatedTableIf.java | 29 -- .../main/java/org/apache/doris/mtmv/MTMVUtil.java | 100 +-- .../doris/mtmv/RelatedPartitionDescResult.java | 53 .../java/org/apache/doris/nereids/memo/Memo.java | 46 .../doris/nereids/parser/LogicalPlanBuilder.java | 24 +- .../mv/AbstractMaterializedViewRule.java | 49 +++- .../mv/LogicalCompatibilityContext.java
(doris) 02/18: fix backup and restore failed between force_replication_allocation setted and not setted clusters (#34608)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f7801948ad81d968f77d7be9835d64d8fedc5cca
Author: camby
AuthorDate: Tue May 14 10:13:28 2024 +0800
fix backup and restore failed between force_replication_allocation setted
and not setted clusters (#34608)
---
fe/fe-core/src/main/java/org/apache/doris/analysis/RestoreStmt.java | 5 +
fe/fe-core/src/main/java/org/apache/doris/backup/BackupJobInfo.java | 4
fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java| 4
3 files changed, 13 insertions(+)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/RestoreStmt.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/RestoreStmt.java
index 9f5f6ee7253..21e417631dd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/RestoreStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/RestoreStmt.java
@@ -20,6 +20,7 @@ package org.apache.doris.analysis;
import org.apache.doris.backup.Repository;
import org.apache.doris.catalog.ReplicaAllocation;
import org.apache.doris.common.AnalysisException;
+import org.apache.doris.common.Config;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.UserException;
@@ -168,6 +169,10 @@ public class RestoreStmt extends AbstractBackupStmt {
ErrorReport.reportAnalysisException(ErrorCode.ERR_COMMON_ERROR,
"Invalid reserve_replica value: " +
copiedProperties.get(PROP_RESERVE_REPLICA));
}
+// force set reserveReplica to false, do not keep the origin
allocation
+if (reserveReplica &&
!Config.force_olap_table_replication_allocation.isEmpty()) {
+reserveReplica = false;
+}
copiedProperties.remove(PROP_RESERVE_REPLICA);
}
// reserve dynamic partition enable
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/backup/BackupJobInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/backup/BackupJobInfo.java
index 7108315cee7..aa127961e3d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/backup/BackupJobInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/backup/BackupJobInfo.java
@@ -32,6 +32,7 @@ import org.apache.doris.catalog.Table;
import org.apache.doris.catalog.TableIf.TableType;
import org.apache.doris.catalog.Tablet;
import org.apache.doris.catalog.View;
+import org.apache.doris.common.Config;
import org.apache.doris.common.FeConstants;
import org.apache.doris.common.Version;
import org.apache.doris.common.io.Text;
@@ -99,6 +100,8 @@ public class BackupJobInfo implements Writable {
public int minorVersion;
@SerializedName("patch_version")
public int patchVersion;
+@SerializedName("is_force_replication_allocation")
+public boolean isForceReplicationAllocation;
@SerializedName("tablet_be_map")
public Map tabletBeMap = Maps.newHashMap();
@@ -607,6 +610,7 @@ public class BackupJobInfo implements Writable {
jobInfo.majorVersion = Version.DORIS_BUILD_VERSION_MAJOR;
jobInfo.minorVersion = Version.DORIS_BUILD_VERSION_MINOR;
jobInfo.patchVersion = Version.DORIS_BUILD_VERSION_PATCH;
+jobInfo.isForceReplicationAllocation =
!Config.force_olap_table_replication_allocation.isEmpty();
Collection tbls = backupMeta.getTables().values();
// tbls
diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java
b/fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java
index 5eb4950466f..2af0242db33 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/backup/RestoreJob.java
@@ -189,6 +189,10 @@ public class RestoreJob extends AbstractJob {
this.state = RestoreJobState.PENDING;
this.metaVersion = metaVersion;
this.reserveReplica = reserveReplica;
+// if backup snapshot is come from a cluster with force replication
allocation, ignore the origin allocation
+if (jobInfo.isForceReplicationAllocation) {
+this.reserveReplica = false;
+}
this.reserveDynamicPartitionEnable = reserveDynamicPartitionEnable;
this.isBeingSynced = isBeingSynced;
properties.put(PROP_RESERVE_REPLICA, String.valueOf(reserveReplica));
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 10/18: [fix](nereids)4 phase agg may lost parameter in some case (#34816)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 8264078a9a8e38f83f89b14ef2e10e6298c48804
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Wed May 15 17:54:30 2024 +0800
[fix](nereids)4 phase agg may lost parameter in some case (#34816)
---
.../apache/doris/nereids/rules/implementation/AggregateStrategies.java | 2 +-
regression-test/suites/nereids_syntax_p0/agg_4_phase.groovy | 2 ++
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/AggregateStrategies.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/AggregateStrategies.java
index edbd28677b4..e1095df7bab 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/AggregateStrategies.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/AggregateStrategies.java
@@ -1766,7 +1766,7 @@ public class AggregateStrategies implements
ImplementationRuleFactory {
AggregateFunction nonDistinct =
aggregateFunction
.withDistinctAndChildren(false,
ImmutableList.copyOf(aggChild));
AggregateExpression nonDistinctAggExpr = new
AggregateExpression(nonDistinct,
-distinctLocalParam,
aggregateFunction.child(0));
+distinctLocalParam, aggregateFunction);
return nonDistinctAggExpr;
} else {
needUpdateSlot.add(aggregateFunction);
diff --git a/regression-test/suites/nereids_syntax_p0/agg_4_phase.groovy
b/regression-test/suites/nereids_syntax_p0/agg_4_phase.groovy
index 4a3e60afb73..19cac99c153 100644
--- a/regression-test/suites/nereids_syntax_p0/agg_4_phase.groovy
+++ b/regression-test/suites/nereids_syntax_p0/agg_4_phase.groovy
@@ -56,4 +56,6 @@ suite("agg_4_phase") {
contains ":VAGGREGATE (update serialize)"
}
qt_4phase (test_sql)
+
+sql """select GROUP_CONCAT(distinct name, " ") from agg_4_phase_tbl;"""
}
\ No newline at end of file
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 04/18: [opt](mtmv) Cache materialization check result for performance (#34301)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 38bac76b376c317a536a5d4b604594107dbea6f5
Author: seawinde <149132972+seawi...@users.noreply.github.com>
AuthorDate: Tue May 14 10:51:15 2024 +0800
[opt](mtmv) Cache materialization check result for performance (#34301)
Need check materialization sql pattern in different abstract rule when
rewrite by materialized view.
Such as the subClass of AbstractMaterializedViewJoinRule,
MaterializedViewScanRule, AbstractMaterializedViewAggregateRule.
This check result can be cached when has checked, this can avoid
unnecessary repeat check
---
.../java/org/apache/doris/nereids/memo/Memo.java | 46 ++
.../mv/AbstractMaterializedViewRule.java | 40 ---
2 files changed, 81 insertions(+), 5 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java
index d7d46ecc15e..09a0ca69840 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/memo/Memo.java
@@ -30,6 +30,7 @@ import org.apache.doris.nereids.properties.LogicalProperties;
import org.apache.doris.nereids.properties.PhysicalProperties;
import org.apache.doris.nereids.properties.RequestPropertyDeriver;
import org.apache.doris.nereids.properties.RequirePropertiesSupplier;
+import
org.apache.doris.nereids.rules.exploration.mv.AbstractMaterializedViewRule;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.plans.GroupPlan;
import org.apache.doris.nereids.trees.plans.LeafPlan;
@@ -53,6 +54,7 @@ import org.apache.logging.log4j.Logger;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;
@@ -74,6 +76,10 @@ public class Memo {
private static long stateId = 0;
private final ConnectContext connectContext;
private final AtomicLong refreshVersion = new AtomicLong(1);
+private final Map,
Set> materializationCheckSuccessMap =
+new LinkedHashMap<>();
+private final Map,
Set> materializationCheckFailMap =
+new LinkedHashMap<>();
private final IdGenerator groupIdGenerator =
GroupId.createGenerator();
private final Map groups = Maps.newLinkedHashMap();
// we could not use Set, because Set does not have get method.
@@ -127,6 +133,46 @@ public class Memo {
return refreshVersion.get();
}
+/**
+ * Record materialization check result for performance
+ */
+public void recordMaterializationCheckResult(Class target,
+Long checkedMaterializationId, boolean isSuccess) {
+if (isSuccess) {
+Set checkedSet = materializationCheckSuccessMap.get(target);
+if (checkedSet == null) {
+checkedSet = new HashSet<>();
+materializationCheckSuccessMap.put(target, checkedSet);
+}
+checkedSet.add(checkedMaterializationId);
+} else {
+Set checkResultSet = materializationCheckFailMap.get(target);
+if (checkResultSet == null) {
+checkResultSet = new HashSet<>();
+materializationCheckFailMap.put(target, checkResultSet);
+}
+checkResultSet.add(checkedMaterializationId);
+}
+}
+
+/**
+ * Get the info for materialization context is checked
+ *
+ * @return if true, check successfully, if false check fail, if null not
checked
+ */
+public Boolean materializationHasChecked(Class target,
+long materializationId) {
+Set checkSuccessSet = materializationCheckSuccessMap.get(target);
+if (checkSuccessSet != null &&
checkSuccessSet.contains(materializationId)) {
+return true;
+}
+Set checkFailSet = materializationCheckFailMap.get(target);
+if (checkFailSet != null && checkFailSet.contains(materializationId)) {
+return false;
+}
+return null;
+}
+
private Plan skipProject(Plan plan, Group targetGroup) {
// Some top project can't be eliminated
if (plan instanceof LogicalProject && ((LogicalProject)
plan).canEliminate()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
index dcff9db69db..41157a6c863 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMa
(doris) 05/18: [fix](group_commit) Wal reader should check block length to avoid reading empty block (#34792)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 6b1c441258a237185769da69a7d0c32e0a2b3c8d
Author: huanghaibin <284824...@qq.com>
AuthorDate: Tue May 14 16:48:27 2024 +0800
[fix](group_commit) Wal reader should check block length to avoid reading
empty block (#34792)
---
be/src/olap/wal/wal_reader.cpp| 3 +++
be/src/vec/exec/format/wal/wal_reader.cpp | 5 +
2 files changed, 8 insertions(+)
diff --git a/be/src/olap/wal/wal_reader.cpp b/be/src/olap/wal/wal_reader.cpp
index 9e4618b2bc1..fa96f0c5a0b 100644
--- a/be/src/olap/wal/wal_reader.cpp
+++ b/be/src/olap/wal/wal_reader.cpp
@@ -69,6 +69,9 @@ Status WalReader::read_block(PBlock& block) {
file_reader->read_at(_offset, {row_len_buf,
WalWriter::LENGTH_SIZE}, &bytes_read));
_offset += WalWriter::LENGTH_SIZE;
size_t block_len = decode_fixed64_le(row_len_buf);
+if (block_len == 0) {
+return Status::DataQualityError("fail to read wal {} ,block is empty",
_file_name);
+}
// read block
std::string block_buf;
block_buf.resize(block_len);
diff --git a/be/src/vec/exec/format/wal/wal_reader.cpp
b/be/src/vec/exec/format/wal/wal_reader.cpp
index 0d15c198b0e..5010f1912ab 100644
--- a/be/src/vec/exec/format/wal/wal_reader.cpp
+++ b/be/src/vec/exec/format/wal/wal_reader.cpp
@@ -68,6 +68,11 @@ Status WalReader::get_next_block(Block* block, size_t*
read_rows, bool* eof) {
LOG(WARNING) << "Failed to read wal on path = " << _wal_path;
return st;
}
+int be_exec_version = pblock.has_be_exec_version() ?
pblock.be_exec_version() : 0;
+if (!BeExecVersionManager::check_be_exec_version(be_exec_version)) {
+return Status::DataQualityError("check be exec version fail when
reading wal file {}",
+_wal_path);
+}
vectorized::Block src_block;
RETURN_IF_ERROR(src_block.deserialize(pblock));
//convert to dst block
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 14/18: [fix](auto inc) db_id and table_id should be int64_t instead of int32_t (#34912)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit dff61715461bf8e6e3736cc4ed14e601c93a3feb
Author: abmdocrt
AuthorDate: Wed May 15 23:00:34 2024 +0800
[fix](auto inc) db_id and table_id should be int64_t instead of int32_t
(#34912)
---
be/src/olap/iterators.h | 2 +-
be/src/olap/rowset/segment_v2/column_reader.h| 2 +-
be/src/olap/rowset/segment_v2/segment_iterator.h | 4 ++--
be/src/olap/schema_cache.cpp | 4 ++--
be/src/olap/schema_cache.h | 4 ++--
be/src/olap/tablet_schema.h | 12 ++--
be/src/olap/utils.h | 4 ++--
be/src/runtime/descriptors.h | 4 ++--
be/src/vec/sink/autoinc_buffer.cpp | 7 +++
9 files changed, 25 insertions(+), 18 deletions(-)
diff --git a/be/src/olap/iterators.h b/be/src/olap/iterators.h
index 95e905185ac..deb14ff554f 100644
--- a/be/src/olap/iterators.h
+++ b/be/src/olap/iterators.h
@@ -115,7 +115,7 @@ public:
RuntimeState* runtime_state = nullptr;
RowsetId rowset_id;
Version version;
-int32_t tablet_id = 0;
+int64_t tablet_id = 0;
// slots that cast may be eliminated in storage layer
std::map target_cast_type_for_variants;
RowRanges row_ranges;
diff --git a/be/src/olap/rowset/segment_v2/column_reader.h
b/be/src/olap/rowset/segment_v2/column_reader.h
index 9ef9d53d662..5d25758e17f 100644
--- a/be/src/olap/rowset/segment_v2/column_reader.h
+++ b/be/src/olap/rowset/segment_v2/column_reader.h
@@ -599,7 +599,7 @@ public:
private:
rowid_t _current_rowid = 0;
-int32_t _tablet_id = 0;
+int64_t _tablet_id = 0;
RowsetId _rowset_id;
int32_t _segment_id = 0;
};
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.h
b/be/src/olap/rowset/segment_v2/segment_iterator.h
index 50c195fb720..23d3d76dfc6 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.h
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.h
@@ -124,7 +124,7 @@ public:
bool is_lazy_materialization_read() const override { return
_lazy_materialization_read; }
uint64_t data_id() const override { return _segment->id(); }
RowsetId rowset_id() const { return _segment->rowset_id(); }
-int32_t tablet_id() const { return _tablet_id; }
+int64_t tablet_id() const { return _tablet_id; }
bool update_profile(RuntimeProfile* profile) override {
bool updated = false;
@@ -486,7 +486,7 @@ private:
// used to collect filter information.
std::vector _filter_info_id;
bool _record_rowids = false;
-int32_t _tablet_id = 0;
+int64_t _tablet_id = 0;
std::set _output_columns;
std::unique_ptr _path_reader;
diff --git a/be/src/olap/schema_cache.cpp b/be/src/olap/schema_cache.cpp
index 7bf6b592c6e..765e95cdc31 100644
--- a/be/src/olap/schema_cache.cpp
+++ b/be/src/olap/schema_cache.cpp
@@ -40,7 +40,7 @@ SchemaCache* SchemaCache::instance() {
}
// format: tabletId-unique_id1-uniqueid2...-version-type
-std::string SchemaCache::get_schema_key(int32_t tablet_id, const
TabletSchemaSPtr& schema,
+std::string SchemaCache::get_schema_key(int64_t tablet_id, const
TabletSchemaSPtr& schema,
const std::vector&
column_ids, int32_t version,
Type type) {
if (column_ids.empty() || schema->column(column_ids[0]).unique_id() < 0) {
@@ -57,7 +57,7 @@ std::string SchemaCache::get_schema_key(int32_t tablet_id,
const TabletSchemaSPt
}
// format: tabletId-unique_id1-uniqueid2...-version-type
-std::string SchemaCache::get_schema_key(int32_t tablet_id, const
std::vector& columns,
+std::string SchemaCache::get_schema_key(int64_t tablet_id, const
std::vector& columns,
int32_t version, Type type) {
if (columns.empty() || columns[0].col_unique_id < 0) {
return "";
diff --git a/be/src/olap/schema_cache.h b/be/src/olap/schema_cache.h
index b94fcf8ccdb..047132e6568 100644
--- a/be/src/olap/schema_cache.h
+++ b/be/src/olap/schema_cache.h
@@ -53,10 +53,10 @@ public:
static void create_global_instance(size_t capacity);
// get cache schema key, delimiter with SCHEMA_DELIMITER
-static std::string get_schema_key(int32_t tablet_id, const
TabletSchemaSPtr& schema,
+static std::string get_schema_key(int64_t tablet_id, const
TabletSchemaSPtr& schema,
const std::vector& column_ids,
int32_t version,
Type type);
-static std::string get_schema_key(int32_t tablet_id, const
std::vector& columns,
+static std::string get_schema_key(int64_t tablet_id, const
std::vector& columns,
int32_t version, Type type);
// Get a shared cached schema fr
(doris) 09/18: [fix](planner) fix date_xxx functions without complete function signature (#34761)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5719f6ff0c6119b752fef130d023d0309571022f
Author: LiBinfeng <46676950+libinfeng...@users.noreply.github.com>
AuthorDate: Wed May 15 16:39:42 2024 +0800
[fix](planner) fix date_xxx functions without complete function signature
(#34761)
Problem:
When using current_date as input of functions like date_sub,
fold constant would failed cause of missing of function signature in Planner
Solved:
Add complete function signature of functions like date_sub
---
.../java/org/apache/doris/rewrite/FEFunctions.java | 68 ++
.../datetime_functions/test_date_function.groovy | 45 ++
2 files changed, 113 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/rewrite/FEFunctions.java
b/fe/fe-core/src/main/java/org/apache/doris/rewrite/FEFunctions.java
index f5d03d68b26..520cdeefff0 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/rewrite/FEFunctions.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/rewrite/FEFunctions.java
@@ -104,6 +104,11 @@ public class FEFunctions {
return daysAdd(date, day);
}
+@FEFunction(name = "date_add", argTypes = { "DATE", "INT" }, returnType =
"DATE")
+public static DateLiteral dateAddDate(LiteralExpr date, LiteralExpr day)
throws AnalysisException {
+return daysAdd(date, day);
+}
+
@FEFunction(name = "date_add", argTypes = { "DATEV2", "INT" }, returnType
= "DATEV2")
public static DateLiteral dateAddDateV2(LiteralExpr date, LiteralExpr day)
throws AnalysisException {
return daysAdd(date, day);
@@ -135,6 +140,12 @@ public class FEFunctions {
return dateLiteral.plusYears(year.getLongValue());
}
+@FEFunction(name = "years_add", argTypes = { "DATE", "INT" }, returnType =
"DATE")
+public static DateLiteral yearsAddDate(LiteralExpr date, LiteralExpr year)
throws AnalysisException {
+DateLiteral dateLiteral = (DateLiteral) date;
+return dateLiteral.plusYears((int) year.getLongValue());
+}
+
@FEFunction(name = "years_add", argTypes = { "DATEV2", "INT" }, returnType
= "DATEV2")
public static DateLiteral yearsAddDateV2(LiteralExpr date, LiteralExpr
year) throws AnalysisException {
DateLiteral dateLiteral = (DateLiteral) date;
@@ -153,6 +164,12 @@ public class FEFunctions {
return dateLiteral.plusMonths(month.getLongValue());
}
+@FEFunction(name = "months_add", argTypes = { "DATE", "INT" }, returnType
= "DATE")
+public static DateLiteral monthsAddDate(LiteralExpr date, LiteralExpr
month) throws AnalysisException {
+DateLiteral dateLiteral = (DateLiteral) date;
+return dateLiteral.plusMonths((int) month.getLongValue());
+}
+
@FEFunction(name = "months_add", argTypes = { "DATEV2", "INT" },
returnType = "DATEV2")
public static DateLiteral monthsAddDateV2(LiteralExpr date, LiteralExpr
month) throws AnalysisException {
DateLiteral dateLiteral = (DateLiteral) date;
@@ -171,6 +188,12 @@ public class FEFunctions {
return dateLiteral.plusDays(day.getLongValue());
}
+@FEFunction(name = "days_add", argTypes = { "DATE", "INT" }, returnType =
"DATE")
+public static DateLiteral daysAddDate(LiteralExpr date, LiteralExpr day)
throws AnalysisException {
+DateLiteral dateLiteral = (DateLiteral) date;
+return dateLiteral.plusDays((int) day.getLongValue());
+}
+
@FEFunction(name = "days_add", argTypes = { "DATEV2", "INT" }, returnType
= "DATEV2")
public static DateLiteral daysAddDateV2(LiteralExpr date, LiteralExpr day)
throws AnalysisException {
DateLiteral dateLiteral = (DateLiteral) date;
@@ -224,26 +247,71 @@ public class FEFunctions {
return dateAdd(date, new IntLiteral(-(int) day.getLongValue()));
}
+@FEFunction(name = "date_sub", argTypes = { "DATE", "INT" }, returnType =
"DATE")
+public static DateLiteral dateSubDate(LiteralExpr date, LiteralExpr day)
throws AnalysisException {
+return daysAdd(date, new IntLiteral(-(int) day.getLongValue()));
+}
+
@FEFunction(name = "date_sub", argTypes = { "DATEV2", "INT" }, returnType
= "DATEV2")
public static DateLiteral dateSubDateV2(LiteralExpr date, LiteralExpr day)
throws AnalysisException {
return dateAdd(date, new IntLiteral(-(int) day.getLongValue()));
}
+@FEFunction(name = "date_sub", argTypes = { "DATETIMEV2", "INT" },
returnType = "DATETIMEV2")
+public static DateLiteral dateSubDateTimeV2(LiteralExpr date, LiteralExpr
day) throws AnalysisException {
+return daysAdd(date, new IntLiteral(-(int) day.getLongValue()));
+}
+
@FEFunction(name = "years_sub", argTypes = { "DATETIME", "INT" },
returnType = "DATETIME")
public static DateLiteral yearsSub(Literal
(doris) 15/18: [Fix](inverted index) fix wrong fs in inverted_index_file_writer (#34903)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 0febfc10e4548ea704d3e3f54029996525d8f107
Author: airborne12
AuthorDate: Wed May 15 23:19:26 2024 +0800
[Fix](inverted index) fix wrong fs in inverted_index_file_writer (#34903)
---
be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp
b/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp
index 9714480318e..5727a452bf1 100644
--- a/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp
+++ b/be/src/olap/rowset/segment_v2/inverted_index_file_writer.cpp
@@ -52,7 +52,7 @@ Result InvertedIndexFileWriter::open(const
TabletIndex* index
(_index_file_dir / _segment_file_name).native(), index_id,
index_suffix);
bool exists = false;
-auto st = _fs->exists(lfs_index_path.c_str(), &exists);
+auto st = _lfs->exists(lfs_index_path.c_str(), &exists);
if (!st.ok()) {
LOG(ERROR) << "index_path:" << lfs_index_path << " exists error:" <<
st;
return ResultError(st);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 16/18: [fix](case)fix PolicyTest testMergeFilterNereidsPlanner failed (#34637)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5d1f5968ebe460eb93aee2c2fca1ae02371fecce
Author: zhangdong <493738...@qq.com>
AuthorDate: Thu May 16 16:42:54 2024 +0800
[fix](case)fix PolicyTest testMergeFilterNereidsPlanner failed (#34637)
The results of explain may have some order that is not fixed
---
fe/fe-core/src/test/java/org/apache/doris/policy/PolicyTest.java | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/fe/fe-core/src/test/java/org/apache/doris/policy/PolicyTest.java
b/fe/fe-core/src/test/java/org/apache/doris/policy/PolicyTest.java
index 3169a57159f..ddf0d68fd2b 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/policy/PolicyTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/policy/PolicyTest.java
@@ -283,7 +283,10 @@ public class PolicyTest extends TestWithFeService {
createPolicy("CREATE ROW POLICY test_row_policy4 ON test.table1 AS
PERMISSIVE TO test_policy USING (k2 = 1)");
String queryStr = "EXPLAIN select * from test.table1";
String explainString = getSQLPlanOrErrorMsg(queryStr);
-Assertions.assertTrue(explainString.contains("k2[#1] IN (1, 2) AND
(k1[#0] = 1)"));
+System.out.println(explainString);
+Assertions.assertTrue(explainString.contains("IN (1, 2)") ||
explainString.contains("IN (2, 1)"));
+Assertions.assertTrue(explainString.contains("AND"));
+Assertions.assertTrue(explainString.contains("= 1)"));
dropPolicy("DROP ROW POLICY test_row_policy1 ON test.table1");
dropPolicy("DROP ROW POLICY test_row_policy3 ON test.table1");
dropPolicy("DROP ROW POLICY test_row_policy4 ON test.table1");
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 17/18: [fix](txn insert) Fix txn insert values error when connect to follower fe (#34950)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit b51a4212d682ffbfb1f789290d777edce8707037
Author: meiyi
AuthorDate: Thu May 16 17:54:39 2024 +0800
[fix](txn insert) Fix txn insert values error when connect to follower fe
(#34950)
---
.../insert/BatchInsertIntoTableCommand.java| 4 +--
.../trees/plans/commands/insert/InsertUtils.java | 29 --
2 files changed, 23 insertions(+), 10 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BatchInsertIntoTableCommand.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BatchInsertIntoTableCommand.java
index 6d4431eec27..4399cd57db4 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BatchInsertIntoTableCommand.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/BatchInsertIntoTableCommand.java
@@ -33,7 +33,7 @@ import org.apache.doris.nereids.trees.plans.Explainable;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.PlanType;
import org.apache.doris.nereids.trees.plans.commands.Command;
-import org.apache.doris.nereids.trees.plans.commands.ForwardWithSync;
+import org.apache.doris.nereids.trees.plans.commands.NoForward;
import org.apache.doris.nereids.trees.plans.logical.LogicalInlineTable;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
import org.apache.doris.nereids.trees.plans.physical.PhysicalOlapTableSink;
@@ -59,7 +59,7 @@ import java.util.stream.Collectors;
/**
* insert into values with in txn model.
*/
-public class BatchInsertIntoTableCommand extends Command implements
ForwardWithSync, Explainable {
+public class BatchInsertIntoTableCommand extends Command implements NoForward,
Explainable {
public static final Logger LOG =
LogManager.getLogger(BatchInsertIntoTableCommand.class);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java
index ad974e9e7bc..8293183eeb2 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java
@@ -24,7 +24,7 @@ import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.Table;
import org.apache.doris.catalog.TableIf;
-import org.apache.doris.common.UserException;
+import org.apache.doris.common.Config;
import org.apache.doris.datasource.hive.HMSExternalTable;
import org.apache.doris.nereids.analyzer.UnboundAlias;
import org.apache.doris.nereids.analyzer.UnboundHiveTableSink;
@@ -54,11 +54,14 @@ import org.apache.doris.nereids.util.TypeCoercionUtils;
import org.apache.doris.proto.InternalService;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.InsertStreamTxnExecutor;
+import org.apache.doris.qe.MasterTxnExecutor;
import org.apache.doris.qe.SessionVariable;
import org.apache.doris.qe.StmtExecutor;
import org.apache.doris.service.FrontendOptions;
import org.apache.doris.thrift.TFileFormatType;
import org.apache.doris.thrift.TFileType;
+import org.apache.doris.thrift.TLoadTxnBeginRequest;
+import org.apache.doris.thrift.TLoadTxnBeginResult;
import org.apache.doris.thrift.TMergeType;
import org.apache.doris.thrift.TStreamLoadPutRequest;
import org.apache.doris.thrift.TTxnParams;
@@ -182,15 +185,25 @@ public class InsertUtils {
txnEntry.setDb(dbObj);
String label = txnEntry.getLabel();
try {
-long txnId = Env.getCurrentGlobalTransactionMgr().beginTransaction(
-txnConf.getDbId(), Lists.newArrayList(tblObj.getId()),
-label, new TransactionState.TxnCoordinator(
-TransactionState.TxnSourceType.FE,
FrontendOptions.getLocalHostAddress()),
-sourceType, timeoutSecond);
-txnConf.setTxnId(txnId);
+long txnId;
String token =
Env.getCurrentEnv().getLoadManager().getTokenManager().acquireToken();
+if (Config.isCloudMode() || Env.getCurrentEnv().isMaster()) {
+txnId = Env.getCurrentGlobalTransactionMgr().beginTransaction(
+txnConf.getDbId(), Lists.newArrayList(tblObj.getId()),
+label, new TransactionState.TxnCoordinator(
+TransactionState.TxnSourceType.FE,
FrontendOptions.getLocalHostAddress()),
+sourceType, timeoutSecond);
+} else {
+MasterTxnExecutor masterTxnExecutor = new
MasterTxnExecutor(ctx);
+
(doris) 01/18: [fix](group commit) Make compatibility issues on serializing and deserializing wal file more clear (#34793)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 6c515e0c76a38116332e998ec1650228941d6d3b
Author: huanghaibin <284824...@qq.com>
AuthorDate: Tue May 14 09:50:29 2024 +0800
[fix](group commit) Make compatibility issues on serializing and
deserializing wal file more clear (#34793)
---
be/src/agent/be_exec_version_manager.h| 1 +
be/src/vec/exec/format/wal/wal_reader.cpp | 6 --
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/be/src/agent/be_exec_version_manager.h
b/be/src/agent/be_exec_version_manager.h
index 32a520cc4bd..248bcfc7e1a 100644
--- a/be/src/agent/be_exec_version_manager.h
+++ b/be/src/agent/be_exec_version_manager.h
@@ -77,5 +77,6 @@ constexpr inline int
BeExecVersionManager::min_be_exec_version = 0;
/// functional
constexpr inline int BITMAP_SERDE = 3;
constexpr inline int USE_NEW_SERDE = 4; // release on DORIS version 2.1
+constexpr inline int OLD_WAL_SERDE = 3; // use to solve compatibility issues,
see pr #32299
} // namespace doris
diff --git a/be/src/vec/exec/format/wal/wal_reader.cpp
b/be/src/vec/exec/format/wal/wal_reader.cpp
index 1fbd395ad26..0d15c198b0e 100644
--- a/be/src/vec/exec/format/wal/wal_reader.cpp
+++ b/be/src/vec/exec/format/wal/wal_reader.cpp
@@ -17,6 +17,7 @@
#include "wal_reader.h"
+#include "agent/be_exec_version_manager.h"
#include "common/logging.h"
#include "common/sync_point.h"
#include "gutil/strings/split.h"
@@ -52,9 +53,10 @@ Status WalReader::get_next_block(Block* block, size_t*
read_rows, bool* eof) {
// in the latest version. So if a wal is created by 2.1.0 (wal version=0
&& be_exec_version=3),
// it should upgrade the be_exec_version to 4 to use the new way to
deserialize pblock to solve
// compatibility issues.see https://github.com/apache/doris/pull/32299
-if (_version == 0 && pblock.has_be_exec_version() &&
pblock.be_exec_version() == 3) {
+if (_version == 0 && pblock.has_be_exec_version() &&
+pblock.be_exec_version() == OLD_WAL_SERDE) {
VLOG_DEBUG << "need to set be_exec_version to 4 to solve compatibility
issues";
-pblock.set_be_exec_version(4);
+pblock.set_be_exec_version(USE_NEW_SERDE);
}
if (st.is()) {
LOG(INFO) << "read eof on wal:" << _wal_path;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 06/18: [Improve](inverted index) improve query performance by not using output index result column (#34281)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 1e53a2a81d19e6a1c2b67b33c6b300e34c789ba8
Author: airborne12
AuthorDate: Tue May 14 17:04:24 2024 +0800
[Improve](inverted index) improve query performance by not using output
index result column (#34281)
---
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
index 3dbb8b694ae..a74fb305795 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
@@ -2265,7 +2265,9 @@ Status
SegmentIterator::_next_batch_internal(vectorized::Block* block) {
RETURN_IF_ERROR(_convert_to_expected_type(_first_read_column_ids));
RETURN_IF_ERROR(_convert_to_expected_type(_non_predicate_columns));
_output_non_pred_columns(block);
-_output_index_result_column(nullptr, 0, block);
+if (!_enable_common_expr_pushdown ||
!_remaining_conjunct_roots.empty()) {
+_output_index_result_column(nullptr, 0, block);
+}
} else {
uint16_t selected_size = _current_batch_rows_read;
uint16_t sel_rowid_idx[selected_size];
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 07/18: [Chore](rollup) check duplicate column name when create table with rollup (#34827)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 4a8df535537e8eab8fa2ad54934a185e17d4e660
Author: Pxl
AuthorDate: Wed May 15 10:25:22 2024 +0800
[Chore](rollup) check duplicate column name when create table with rollup
(#34827)
check duplicate column name when create table with rollup
---
.../trees/plans/commands/info/CreateTableInfo.java | 4
.../plans/commands/info/RollupDefinition.java | 14 -
.../test_mv_useless/test_dup_mv_useless.groovy | 23 ++
.../suites/nereids_p0/create_table/ddl/table.sql | 2 +-
4 files changed, 41 insertions(+), 2 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java
index a4cf08efe74..585d1da0b10 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java
@@ -459,6 +459,10 @@ public class CreateTableInfo {
}
}
}
+
+for (RollupDefinition rollup : rollups) {
+rollup.validate();
+}
} else {
// mysql, broker and hive do not need key desc
if (keysType != null) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/RollupDefinition.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/RollupDefinition.java
index 6c3857279f5..b01e380e904 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/RollupDefinition.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/RollupDefinition.java
@@ -18,12 +18,15 @@
package org.apache.doris.nereids.trees.plans.commands.info;
import org.apache.doris.analysis.AddRollupClause;
+import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.util.Utils;
import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
import java.util.List;
import java.util.Map;
+import java.util.Set;
/**
* rollup definition
@@ -41,7 +44,16 @@ public class RollupDefinition {
this.properties = Maps.newHashMap(properties);
}
-public void validate() {
+/**
+ * check rollup validity
+ */
+public void validate() throws AnalysisException {
+Set colSet = Sets.newHashSet();
+for (String col : cols) {
+if (!colSet.add(col)) {
+throw new AnalysisException("rollup has duplicate column name
" + col);
+}
+}
}
public AddRollupClause translateToCatalogStyle() {
diff --git
a/regression-test/suites/mv_p0/test_mv_useless/test_dup_mv_useless.groovy
b/regression-test/suites/mv_p0/test_mv_useless/test_dup_mv_useless.groovy
index 7640b6e1806..d91cafbe93d 100644
--- a/regression-test/suites/mv_p0/test_mv_useless/test_dup_mv_useless.groovy
+++ b/regression-test/suites/mv_p0/test_mv_useless/test_dup_mv_useless.groovy
@@ -48,4 +48,27 @@ suite ("test_dup_mv_useless") {
createMV("create materialized view k1_k2_u21 as select k2,k1 from
${testTable} group by k2,k1 order by k2,k1;")
createMV("create materialized view k1_k2_sumk3 as select k1,k2,sum(k3)
from ${testTable} group by k1,k2;")
sql "insert into ${testTable} select 4,4,4;"
+
+test {
+sql """
+create table test_rollup (
+`id` int not null,
+`kbool` boolean not null,
+`ktint` tinyint(4) not null,
+`ksint` smallint(6) not null,
+`kint` int(11) not null,
+`kbint` bigint(20) not null,
+`klint` largeint(40) not null
+) engine=OLAP
+duplicate key(id, kbool, ktint)
+distributed by random buckets auto
+rollup (
+r1 (id, ktint, kbool, ktint, kbint) duplicate key(id)
+)
+properties (
+"replication_num"="1"
+);
+"""
+exception "duplicate column name"
+}
}
diff --git a/regression-test/suites/nereids_p0/create_table/ddl/table.sql
b/regression-test/suites/nereids_p0/create_table/ddl/table.sql
index bcac1168b73..c1f24407178 100644
--- a/regression-test/suites/nereids_p0/create_table/ddl/table.sql
+++ b/regression-test/suites/nereids_p0/create_table/ddl/table.sql
@@ -238,7 +238,7 @@ create table test_rollup (
duplicate key(id, kbool, ktint)
distributed by random buckets auto
rollup (
-r1 (id, ktint, kbool, ktint, kbint) duplicate key(id)
+r1 (id, ktint, kbool, kbint) duplicate key(id)
)
properties (
"replication_num"="1"
--
(doris) 11/18: [fix](planner)correlated predicate should include isnull predicate (#34833)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit a07876e807d10161537a84c699a09586c3ac9c30
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Wed May 15 18:02:18 2024 +0800
[fix](planner)correlated predicate should include isnull predicate (#34833)
---
.../org/apache/doris/analysis/StmtRewriter.java| 3 +-
.../correctness_p0/test_subquery_with_agg.groovy | 41 ++
2 files changed, 43 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java
index 62737131b62..93823cf398c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java
@@ -957,7 +957,8 @@ public class StmtRewriter {
* query block (i.e. is not bound by the given 'tupleIds').
*/
private static boolean isCorrelatedPredicate(Expr expr, List
tupleIds) {
-return (expr instanceof BinaryPredicate || expr instanceof SlotRef) &&
!expr.isBoundByTupleIds(tupleIds);
+return (expr instanceof BinaryPredicate || expr instanceof SlotRef
+|| expr instanceof IsNullPredicate) &&
!expr.isBoundByTupleIds(tupleIds);
}
/**
diff --git
a/regression-test/suites/correctness_p0/test_subquery_with_agg.groovy
b/regression-test/suites/correctness_p0/test_subquery_with_agg.groovy
index e0592830ffe..a962d64dcbc 100644
--- a/regression-test/suites/correctness_p0/test_subquery_with_agg.groovy
+++ b/regression-test/suites/correctness_p0/test_subquery_with_agg.groovy
@@ -82,4 +82,45 @@ suite("test_subquery_with_agg") {
drop table if exists agg_subquery_table;
"""
+sql """drop table if exists subquery_table_xyz;"""
+sql """CREATE TABLE `subquery_table_xyz` (
+`phone`bigint(20) NULL
+) ENGINE=OLAP
+DUPLICATE KEY(`phone`)
+COMMENT 'OLAP'
+DISTRIBUTED BY HASH(`phone`) BUCKETS 3
+PROPERTIES (
+"replication_allocation" = "tag.location.default: 1"
+);"""
+sql """WITH tmp1 AS
+(SELECT DISTINCT phone
+FROM subquery_table_xyz oua
+WHERE (NOT EXISTS
+(SELECT 1
+FROM subquery_table_xyz o1
+WHERE oua.phone = o1.phone
+AND phone IS NOT NULL))),
+tmp2 AS
+(SELECT DISTINCT phone
+FROM subquery_table_xyz oua
+WHERE (NOT EXISTS
+(SELECT 1
+FROM subquery_table_xyz o1
+WHERE oua.phone = o1.phone
+and phone IS NOT NULL))),
+tmp3 AS
+(SELECT DISTINCT phone
+FROM subquery_table_xyz oua
+WHERE (NOT EXISTS
+(SELECT 1
+FROM subquery_table_xyz o1
+WHERE oua.phone = o1.phone and
+phone IS NOT NULL)))
+SELECT COUNT(DISTINCT tmp1.phone)
+FROM tmp1
+JOIN tmp2
+ON tmp1.phone = tmp2.phone
+JOIN tmp3
+ON tmp2.phone = tmp3.phone;"""
+
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 12/18: [fix](load) skip sending cancel rpc if VNodeChannel is not inited (#34897)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 849eeb39e9c25645498966c681c7d2c115db8262
Author: Kaijie Chen
AuthorDate: Wed May 15 18:26:11 2024 +0800
[fix](load) skip sending cancel rpc if VNodeChannel is not inited (#34897)
---
be/src/vec/sink/writer/vtablet_writer.cpp | 4
1 file changed, 4 insertions(+)
diff --git a/be/src/vec/sink/writer/vtablet_writer.cpp
b/be/src/vec/sink/writer/vtablet_writer.cpp
index c2a029e7870..3fb17850eff 100644
--- a/be/src/vec/sink/writer/vtablet_writer.cpp
+++ b/be/src/vec/sink/writer/vtablet_writer.cpp
@@ -841,6 +841,10 @@ void VNodeChannel::cancel(const std::string& cancel_msg) {
// we don't need to wait last rpc finished, cause closure's release/reset
will join.
// But do we need brpc::StartCancel(call_id)?
_cancel_with_msg(cancel_msg);
+// if not inited, _stub will be nullptr, skip sending cancel rpc
+if (!_inited) {
+return;
+}
auto request = std::make_shared();
request->set_allocated_id(&_parent->_load_id);
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 18/18: [Fix](tablet id) use int64_t instead of int32_t or uint32_t for tablet_id (#34962)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit db273d578f0756fa318f06cc77bdf9f5eed21745
Author: lihangyu <15605149...@163.com>
AuthorDate: Thu May 16 19:56:11 2024 +0800
[Fix](tablet id) use int64_t instead of int32_t or uint32_t for tablet_id
(#34962)
---
be/src/olap/rowset/segment_v2/column_reader.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/olap/rowset/segment_v2/column_reader.h
b/be/src/olap/rowset/segment_v2/column_reader.h
index 5d25758e17f..9c889b848fd 100644
--- a/be/src/olap/rowset/segment_v2/column_reader.h
+++ b/be/src/olap/rowset/segment_v2/column_reader.h
@@ -557,7 +557,7 @@ private:
class RowIdColumnIterator : public ColumnIterator {
public:
RowIdColumnIterator() = delete;
-RowIdColumnIterator(int32_t tid, RowsetId rid, int32_t segid)
+RowIdColumnIterator(int64_t tid, RowsetId rid, int32_t segid)
: _tablet_id(tid), _rowset_id(rid), _segment_id(segid) {}
Status seek_to_first() override {
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 08/18: [opt](mtmv) Optimize the logic of slot mapping generate for performance (#34597)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 71caf88ec1563e01329fa0202f2a7cf854d94f6d
Author: seawinde <149132972+seawi...@users.noreply.github.com>
AuthorDate: Wed May 15 13:57:28 2024 +0800
[opt](mtmv) Optimize the logic of slot mapping generate for performance
(#34597)
Slot mapping is used for materialized view rewritting
given the relation mapping, the slot mapping is the same
Optimize the slot mapping genarate logic
Cache the slot mapping in materialization context by realation mapping key
---
.../mv/AbstractMaterializedViewRule.java | 9 +++-
.../mv/LogicalCompatibilityContext.java| 7 +--
.../exploration/mv/MaterializationContext.java | 13 ++
.../exploration/mv/mapping/ExpressionMapping.java | 23 --
.../rules/exploration/mv/mapping/Mapping.java | 24 +++
.../exploration/mv/mapping/RelationMapping.java| 18
.../rules/exploration/mv/mapping/SlotMapping.java | 50 +-
7 files changed, 86 insertions(+), 58 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
index 41157a6c863..a88362e6697 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
@@ -179,7 +179,12 @@ public abstract class AbstractMaterializedViewRule
implements ExplorationRuleFac
return rewriteResults;
}
for (RelationMapping queryToViewTableMapping :
queryToViewTableMappings) {
-SlotMapping queryToViewSlotMapping =
SlotMapping.generate(queryToViewTableMapping);
+SlotMapping queryToViewSlotMapping =
+
materializationContext.getSlotMappingFromCache(queryToViewTableMapping);
+if (queryToViewSlotMapping == null) {
+queryToViewSlotMapping =
SlotMapping.generate(queryToViewTableMapping);
+
materializationContext.addSlotMappingToCache(queryToViewTableMapping,
queryToViewSlotMapping);
+}
if (queryToViewSlotMapping == null) {
materializationContext.recordFailReason(queryStructInfo,
"Query to view slot mapping is null", () -> "");
@@ -187,7 +192,7 @@ public abstract class AbstractMaterializedViewRule
implements ExplorationRuleFac
}
SlotMapping viewToQuerySlotMapping =
queryToViewSlotMapping.inverse();
LogicalCompatibilityContext compatibilityContext =
LogicalCompatibilityContext.from(
-queryToViewTableMapping, queryToViewSlotMapping,
queryStructInfo, viewStructInfo);
+queryToViewTableMapping, viewToQuerySlotMapping,
queryStructInfo, viewStructInfo);
ComparisonResult comparisonResult =
StructInfo.isGraphLogicalEquals(queryStructInfo, viewStructInfo,
compatibilityContext);
if (comparisonResult.isInvalid()) {
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/LogicalCompatibilityContext.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/LogicalCompatibilityContext.java
index b4ed509f300..25bafeb64c1 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/LogicalCompatibilityContext.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/LogicalCompatibilityContext.java
@@ -128,7 +128,7 @@ public class LogicalCompatibilityContext {
* this make expression mapping between query and view by relation and the
slot in relation mapping
*/
public static LogicalCompatibilityContext from(RelationMapping
relationMapping,
-SlotMapping queryToViewSlotMapping,
+SlotMapping viewToQuerySlotMapping,
StructInfo queryStructInfo,
StructInfo viewStructInfo) {
// init node mapping
@@ -147,11 +147,8 @@ public class LogicalCompatibilityContext {
queryToViewNodeMapping.put(queryStructInfoNode,
viewStructInfoNode);
}
}
-// init expression mapping
-Map viewToQuerySlotMapping =
queryToViewSlotMapping.inverse()
-.toSlotReferenceMap();
return new LogicalCompatibilityContext(queryToViewNodeMapping,
-viewToQuerySlotMapping,
+viewToQuerySlotMapping.toSlotReferenceMap(),
queryStructInfo,
viewStructInfo);
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/MaterializationContext.ja
(doris) 03/18: [feature](mtmv) create mtmv support partitions rollup (#31812)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a commit to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git commit 30a036e7a4862782076db186934d64f5e4aef3a9 Author: zhangdong <493738...@qq.com> AuthorDate: Tue May 14 10:46:29 2024 +0800 [feature](mtmv) create mtmv support partitions rollup (#31812) if create MTMV `date_trunc(`xxx`,'month')` when related table is `range` partition,and have 3 partitions: ``` 20200101-20200102 20200102-20200103 20200201-20200202 ``` then MTMV will have 2 partitions: ``` 20200101-20200201 20200201-20200301 ``` when related table is `list` partition,and have 3 partitions: ``` (20200101,20200102) (20200103) (20200201) ``` then MTMV will have 2 partitions: ``` (20200101,20200102，20200103) (20200201) ``` --- .../antlr4/org/apache/doris/nereids/DorisParser.g4 | 7 +- .../main/java/org/apache/doris/catalog/MTMV.java | 40 +-- .../apache/doris/job/extensions/mtmv/MTMVTask.java | 6 +- .../doris/mtmv/MTMVPartitionExprDateTrunc.java | 228 .../doris/mtmv/MTMVPartitionExprFactory.java | 39 +++ .../doris/mtmv/MTMVPartitionExprService.java | 60 + .../org/apache/doris/mtmv/MTMVPartitionInfo.java | 39 ++- .../org/apache/doris/mtmv/MTMVPartitionUtil.java | 79 +++--- .../MTMVRelatedPartitionDescGeneratorService.java | 38 +++ .../MTMVRelatedPartitionDescInitGenerator.java | 34 +++ ...latedPartitionDescOnePartitionColGenerator.java | 67 + .../MTMVRelatedPartitionDescRollUpGenerator.java | 142 ++ ...TMVRelatedPartitionDescSyncLimitGenerator.java} | 145 +++--- .../org/apache/doris/mtmv/MTMVRelatedTableIf.java | 29 -- .../main/java/org/apache/doris/mtmv/MTMVUtil.java | 100 +-- .../doris/mtmv/RelatedPartitionDescResult.java | 53 .../doris/nereids/parser/LogicalPlanBuilder.java | 24 +- .../trees/plans/commands/info/CreateMTMVInfo.java | 86 ++ .../commands/info/MTMVPartitionDefinition.java | 215 +++ ...TMVRelatedPartitionDescRollUpGeneratorTest.java | 152 +++ ...elatedPartitionDescSyncLimitGeneratorTest.java} | 45 +--- .../java/org/apache/doris/mtmv/MTMVUtilTest.java | 65 - .../mtmv_p0/test_hive_limit_partition_mtmv.out | 14 + .../data/mtmv_p0/test_rollup_partition_mtmv.out| 16 ++ .../mtmv_p0/test_hive_limit_partition_mtmv.groovy | 129 + .../mtmv_p0/test_rollup_partition_mtmv.groovy | 292 + 26 files changed, 1628 insertions(+), 516 deletions(-) diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 index 9562bed5f21..c78f11b0007 100644 --- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 +++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 @@ -101,7 +101,7 @@ statementBase (REFRESH refreshMethod? refreshTrigger?)? ((DUPLICATE)? KEY keys=identifierList)? (COMMENT STRING_LITERAL)? -(PARTITION BY LEFT_PAREN partitionKey = identifier RIGHT_PAREN)? +(PARTITION BY LEFT_PAREN mvPartition RIGHT_PAREN)? (DISTRIBUTED BY (HASH hashKeys=identifierList | RANDOM) (BUCKETS (INTEGER_VALUE | AUTO))?)? propertyClause? AS query #createMTMV @@ -225,6 +225,11 @@ refreshMethod : COMPLETE | AUTO ; +mvPartition +: partitionKey = identifier +| partitionExpr = functionCallExpression +; + identifierOrStringLiteral : identifier | STRING_LITERAL diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java index c76f1a253f2..a3050bc3ac6 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java @@ -30,6 +30,7 @@ import org.apache.doris.mtmv.MTMVJobInfo; import org.apache.doris.mtmv.MTMVJobManager; import org.apache.doris.mtmv.MTMVPartitionInfo; import org.apache.doris.mtmv.MTMVPartitionInfo.MTMVPartitionType; +import org.apache.doris.mtmv.MTMVPartitionUtil; import org.apache.doris.mtmv.MTMVPlanUtil; import org.apache.doris.mtmv.MTMVRefreshEnum.MTMVRefreshState; import org.apache.doris.mtmv.MTMVRefreshEnum.MTMVState; @@ -38,7 +39,6 @@ import org.apache.doris.mtmv.MTMVRefreshPartitionSnapshot; import org.apache.doris.mtmv.MTMVRefreshSnapshot; import org.apache.doris.mtmv.MTMVRelation; import org.apache.doris.mtmv.MTMVStatus; -import org.apache.doris.mtmv.MTMVUtil; import org.apache.doris.persist.gson.GsonUtils; import com.google.common.collect.Maps; @@ -51,7 +51,6 @@ import org.apache.logging.log4j.Logger; import java.io.DataInput; import java.io.DataOutput; import java.io.IOException; -import java.uti
(doris) 13/18: [fix](move-memtable) change brpc connection type to single (#34883)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 4b96f9834fd213e7baedec78fea9757eedc386d3
Author: Kaijie Chen
AuthorDate: Wed May 15 18:26:41 2024 +0800
[fix](move-memtable) change brpc connection type to single (#34883)
---
be/src/util/brpc_client_cache.h | 6 +-
be/src/vec/sink/load_stream_stub.cpp | 6 +++---
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/be/src/util/brpc_client_cache.h b/be/src/util/brpc_client_cache.h
index 2e1d0508ba3..290f2cc3e04 100644
--- a/be/src/util/brpc_client_cache.h
+++ b/be/src/util/brpc_client_cache.h
@@ -111,7 +111,8 @@ public:
std::shared_ptr get_new_client_no_cache(const std::string& host_port,
const std::string& protocol =
"baidu_std",
- const std::string& connect_type
= "") {
+ const std::string& connect_type
= "",
+ const std::string&
connection_group = "") {
brpc::ChannelOptions options;
if constexpr (std::is_same_v) {
options.protocol = config::function_service_protocol;
@@ -121,6 +122,9 @@ public:
if (connect_type != "") {
options.connection_type = connect_type;
}
+if (connection_group != "") {
+options.connection_group = connection_group;
+}
options.connect_timeout_ms = 2000;
options.max_retry = 10;
diff --git a/be/src/vec/sink/load_stream_stub.cpp
b/be/src/vec/sink/load_stream_stub.cpp
index 155ce2de349..92670c1c930 100644
--- a/be/src/vec/sink/load_stream_stub.cpp
+++ b/be/src/vec/sink/load_stream_stub.cpp
@@ -175,9 +175,9 @@ Status
LoadStreamStub::open(BrpcClientCache* client_cache,
*request.add_tablets() = tablet;
}
POpenLoadStreamResponse response;
-// use "pooled" connection to avoid conflicts between streaming rpc and
regular rpc,
-// see: https://github.com/apache/brpc/issues/392
-const auto& stub = client_cache->get_new_client_no_cache(host_port,
"baidu_std", "pooled");
+// set connection_group "streaming" to distinguish with non-streaming
connections
+const auto& stub =
+client_cache->get_new_client_no_cache(host_port, "baidu_std",
"single", "streaming");
stub->open_load_stream(&cntl, &request, &response, nullptr);
for (const auto& resp : response.tablet_schemas()) {
auto tablet_schema = std::make_unique();
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch branch-2.1 updated (db273d578f0 -> 22f85be7126)
This is an automated email from the ASF dual-hosted git repository. yiguolei pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/doris.git from db273d578f0 [Fix](tablet id) use int64_t instead of int32_t or uint32_t for tablet_id (#34962) new 6f5abfd23fc [regression-test](fix) fix case bug, using test_insert_dft_tbl in multiple test cases #34983 new eb7eaee3862 [fix](function) money format (#34680) new 73419c24311 [enhance](mtmv)MTMV support hive table determine whether the data is sync (#34845) new 5b72dd1217e [chore](test) remove useless drop table in test_list_partition_datatype (#34930) new cc11e50200f [fix](mtmv)Fix slot desc wrong in query rewrite by materialized view when query is complex (#34904) new 05605d99a98 [opt](routine-load) optimize routine load task allocation algorithm (#34778) new 274c96b12db [enhancement](regression-test) modify a key type tests (#34600) new 437c1a1ba40 [enhancement](regression-test) modify a key type tests (#34717) new 9b5028785d1 [fix](prepare) fix datetimev2 return err when binary_row_format (#34662) new e66dd588601 [Improve](inverted index) improve performance by introducing bulk contains for bitmap in _build_index_result_column (#34831) new bfd875eae39 [opt](nereids) lazy get expression map when comparing hypergraph (#34753) new 81bcb9d4909 [opt](planner)(Nereids) support auto aggregation for random distributed table (#33630) new e3e5f18f26a [Fix](Json type) correct cast result for json type (#34764) new 435147d449c [enhance](mtmv) MTMV deal partition use name instead of id (#34910) new a59f9c3fa15 [fix](planner) fix unrequired slot bug when join node introduced by #25204 (#34923) new 89d5f2e816a [fix](multi-catalog)remove http scheme in oss endpoint (#34907) new 22f85be7126 [fix](hive-ctas) support create hive table with full quolified name (#34984) The 17 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../exec/schema_scanner/schema_columns_scanner.cpp | 27 +- be/src/olap/rowset/segment_v2/segment_iterator.cpp | 3 +- be/src/util/jsonb_document.h | 2 +- be/src/util/mysql_row_buffer.cpp | 46 +- be/src/util/mysql_row_buffer.h | 4 +- .../serde/data_type_datetimev2_serde.cpp | 4 +- be/src/vec/functions/function_cast.h | 79 +-- be/src/vec/functions/function_string.h | 180 +++-- be/test/vec/function/function_jsonb_test.cpp | 15 +- .../vec/function/function_money_format_test.cpp| 92 +++ .../org/apache/doris/analysis/StmtRewriter.java| 236 +++ .../main/java/org/apache/doris/catalog/MTMV.java | 30 +- .../java/org/apache/doris/catalog/OlapTable.java | 27 +- .../doris/datasource/hive/HMSExternalTable.java| 38 +- .../datasource/property/PropertyConverter.java | 2 +- .../apache/doris/job/extensions/mtmv/MTMVTask.java | 41 +- .../doris/load/routineload/RoutineLoadManager.java | 18 +- .../doris/mtmv/MTMVMaxTimestampSnapshot.java | 19 +- .../org/apache/doris/mtmv/MTMVPartitionUtil.java | 112 ++- ...latedPartitionDescOnePartitionColGenerator.java | 6 +- .../MTMVRelatedPartitionDescRollUpGenerator.java | 20 +- ...MTMVRelatedPartitionDescSyncLimitGenerator.java | 6 +- .../org/apache/doris/mtmv/MTMVRelatedTableIf.java | 6 +- .../org/apache/doris/mtmv/MTMVRewriteUtil.java | 6 +- .../doris/mtmv/RelatedPartitionDescResult.java | 12 +- .../org/apache/doris/mysql/MysqlSerializer.java| 2 + .../doris/nereids/jobs/executor/Analyzer.java | 3 + .../org/apache/doris/nereids/rules/RuleType.java | 5 +- .../BuildAggForRandomDistributedTable.java | 257 +++ .../mv/AbstractMaterializedViewRule.java | 2 +- .../rules/exploration/mv/HyperGraphComparator.java | 114 ++-- .../mv/InitMaterializationContextHook.java | 5 +- .../mv/LogicalCompatibilityContext.java| 31 +- .../exploration/mv/MaterializationContext.java | 13 +- .../exploration/mv/MaterializedViewUtils.java | 7 +- .../plans/commands/UpdateMvByPartitionCommand.java | 26 +- .../trees/plans/commands/info/CreateTableInfo.java | 17 +- .../trees/plans/visitor/TableCollector.java| 14 +- .../org/apache/doris/planner/JoinNodeBase.java | 6 + .../java/org/apache/doris/qe/StmtExecutor.java | 9 +- .../datasource/property/PropertyConverterTest.java | 4 +- .../apache/doris/mtmv/MTMVPartitionUtilTest.java | 18 +- ...TMVRelatedPartitionDescRollUpGeneratorTest.java | 28 +- .../org/apache/doris/mtmv/MTMVRewriteUtilTest.java | 12 +- .../java/org/apache/doris/mtmv/MTMVTaskTest.java | 46 +- .../test_compactio
(doris) 01/17: [regression-test](fix) fix case bug, using test_insert_dft_tbl in multiple test cases #34983
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 6f5abfd23fcdf959168c7bed76fe7514f3e363f3
Author: shuke <37901441+shuke...@users.noreply.github.com>
AuthorDate: Fri May 17 09:33:47 2024 +0800
[regression-test](fix) fix case bug, using test_insert_dft_tbl in multiple
test cases #34983
---
regression-test/suites/load_p0/insert/test_insert.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/regression-test/suites/load_p0/insert/test_insert.groovy
b/regression-test/suites/load_p0/insert/test_insert.groovy
index 25b010f242d..1ef37f843f7 100644
--- a/regression-test/suites/load_p0/insert/test_insert.groovy
+++ b/regression-test/suites/load_p0/insert/test_insert.groovy
@@ -49,7 +49,7 @@ suite("test_insert") {
qt_sql1 "select * from ${insert_tbl} order by 1, 2, 3, 4"
-def insert_tbl_dft = "test_insert_dft_tbl"
+def insert_tbl_dft = "test_insert_dft2_tbl"
sql """ DROP TABLE IF EXISTS ${insert_tbl_dft}"""
// `k7` should be float type, and bug exists now,
https://github.com/apache/doris/pull/20867
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 04/17: [chore](test) remove useless drop table in test_list_partition_datatype (#34930)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5b72dd1217ee54acd692fea2489caa0a2091757c
Author: starocean999 <40539150+starocean...@users.noreply.github.com>
AuthorDate: Fri May 17 11:24:57 2024 +0800
[chore](test) remove useless drop table in test_list_partition_datatype
(#34930)
---
.../partition_p0/list_partition/test_list_partition_datatype.groovy | 1 -
1 file changed, 1 deletion(-)
diff --git
a/regression-test/suites/partition_p0/list_partition/test_list_partition_datatype.groovy
b/regression-test/suites/partition_p0/list_partition/test_list_partition_datatype.groovy
index 1de3f6588b6..d06a0195adb 100644
---
a/regression-test/suites/partition_p0/list_partition/test_list_partition_datatype.groovy
+++
b/regression-test/suites/partition_p0/list_partition/test_list_partition_datatype.groovy
@@ -499,7 +499,6 @@ suite("test_list_partition_datatype", "p0") {
assertTrue(ret.size() == 1)
try_sql "DROP TABLE IF EXISTS test_list_partition_ddl_tbl_1"
try_sql "DROP TABLE IF EXISTS test_list_partition_empty_tb"
-try_sql "DROP TABLE IF EXISTS test_list_partition_select_tb"
try_sql "DROP TABLE IF EXISTS test_list_partition_tb2_char"
// try_sql "DROP TABLE IF EXISTS test_list_partition_tb3_char"
}
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 02/17: [fix](function) money format (#34680)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit eb7eaee386280e9854863eb03258126d70d99f48
Author: zhiqiang
AuthorDate: Fri May 17 10:44:31 2024 +0800
[fix](function) money format (#34680)
---
be/src/vec/functions/function_string.h | 180 +++--
.../vec/function/function_money_format_test.cpp| 92 +++
.../string_functions/test_string_function.out | 4 +-
.../string_functions/test_money_format.out | 154 ++
.../string_functions/test_money_format.groovy | 77 +
5 files changed, 452 insertions(+), 55 deletions(-)
diff --git a/be/src/vec/functions/function_string.h
b/be/src/vec/functions/function_string.h
index 61339bbd9cf..6dbd2704187 100644
--- a/be/src/vec/functions/function_string.h
+++ b/be/src/vec/functions/function_string.h
@@ -25,15 +25,18 @@
#include
#include
#include
+#include
#include
#include
#include
+#include
#include
#include
#include
#include
#include
#include
+#include
#include
#include
@@ -65,6 +68,7 @@
#include "vec/core/field.h"
#include "vec/core/types.h"
#include "vec/data_types/data_type.h"
+#include "vec/functions/round.h"
#include "vec/io/io_helper.h"
#ifndef USE_LIBCPP
@@ -2959,18 +2963,97 @@ public:
namespace MoneyFormat {
+constexpr size_t MAX_FORMAT_LEN_DEC32() {
+// Decimal(9, 0)
+// Double the size to avoid some unexpected bug.
+return 2 * (1 + 9 + (9 / 3) + 3);
+}
+
+constexpr size_t MAX_FORMAT_LEN_DEC64() {
+// Decimal(18, 0)
+// Double the size to avoid some unexpected bug.
+return 2 * (1 + 18 + (18 / 3) + 3);
+}
+
+constexpr size_t MAX_FORMAT_LEN_DEC128V2() {
+// DecimalV2 has at most 27 digits
+// Double the size to avoid some unexpected bug.
+return 2 * (1 + 27 + (27 / 3) + 3);
+}
+
+constexpr size_t MAX_FORMAT_LEN_DEC128V3() {
+// Decimal(38, 0)
+// Double the size to avoid some unexpected bug.
+return 2 * (1 + 39 + (39 / 3) + 3);
+}
+
+constexpr size_t MAX_FORMAT_LEN_INT64() {
+// INT_MIN = -9223372036854775807
+// Double the size to avoid some unexpected bug.
+return 2 * (1 + 20 + (20 / 3) + 3);
+}
+
+constexpr size_t MAX_FORMAT_LEN_INT128() {
+// INT128_MIN = -170141183460469231731687303715884105728
+return 2 * (1 + 39 + (39 / 3) + 3);
+}
+
template
-StringRef do_money_format(FunctionContext* context, const T int_value,
- const int32_t frac_value = 0) {
+StringRef do_money_format(FunctionContext* context, UInt32 scale, T int_value,
T frac_value) {
+static_assert(std::is_integral::value);
+const bool is_negative = int_value < 0 || frac_value < 0;
+
+// do round to frac_part
+// magic number 2: since we need to round frac_part to 2 digits
+if (scale > 2) {
+DCHECK(scale <= 38);
+// do rounding, so we need to reserve 3 digits.
+auto multiplier = common::exp10_i128(std::abs(static_cast(scale -
3)));
+// do devide first to avoid overflow
+// after round frac_value will be positive by design.
+frac_value = std::abs(frac_value / multiplier) + 5;
+frac_value /= 10;
+} else if (scale < 2) {
+DCHECK(frac_value < 100);
+// since scale <= 2, overflow is impossiable
+frac_value = frac_value * common::exp10_i32(2 - scale);
+}
+
+if (frac_value == 100) {
+if (is_negative) {
+int_value -= 1;
+} else {
+int_value += 1;
+}
+frac_value = 0;
+}
+
+bool append_sign_manually = false;
+if (is_negative && int_value == 0) {
+// when int_value is 0, result of SimpleItoaWithCommas will contains
just zero
+// for Decimal like -0.1234, this will leads to problem, because
negative sign is discarded.
+// this is why we introduce argument append_sing_manually.
+append_sign_manually = true;
+}
+
char local[N];
char* p = SimpleItoaWithCommas(int_value, local, sizeof(local));
-int32_t string_val_len = local + sizeof(local) - p + 3;
-StringRef result = context->create_temp_string_val(string_val_len);
+const Int32 integer_str_len = N - (p - local);
+const Int32 frac_str_len = 2;
+const Int32 whole_decimal_str_len =
+(append_sign_manually ? 1 : 0) + integer_str_len + 1 +
frac_str_len;
+
+StringRef result = context->create_temp_string_val(whole_decimal_str_len);
char* result_data = const_cast(result.data);
-memcpy(result_data, p, string_val_len - 3);
-*(result_data + string_val_len - 3) = '.';
-*(result_data + string_val_len - 2) = '0' + (frac_value / 10);
-*(result_data + string_val_len - 1) = '0' + (frac_value % 10);
+
+if (append_sign_manually) {
+memset(result_data, '-', 1);
+}
+
+memcpy(result_data + (append_sign_manually ? 1 : 0), p, intege
(doris) 03/17: [enhance](mtmv)MTMV support hive table determine whether the data is sync (#34845)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 73419c243116300511f76d1964f7373d540c0673
Author: zhangdong <493738...@qq.com>
AuthorDate: Fri May 17 10:55:10 2024 +0800
[enhance](mtmv)MTMV support hive table determine whether the data is sync
(#34845)
Previously supported, this PR only turns on the switch
---
.../java/org/apache/doris/datasource/hive/HMSExternalTable.java | 2 +-
regression-test/suites/mtmv_p0/test_hive_mtmv.groovy | 8
regression-test/suites/mtmv_p0/test_hive_refresh_mtmv.groovy | 8
3 files changed, 5 insertions(+), 13 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
index c2099a1acc8..57dee5debb0 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java
@@ -958,7 +958,7 @@ public class HMSExternalTable extends ExternalTable
implements MTMVRelatedTableI
@Override
public boolean needAutoRefresh() {
-return false;
+return true;
}
@Override
diff --git a/regression-test/suites/mtmv_p0/test_hive_mtmv.groovy
b/regression-test/suites/mtmv_p0/test_hive_mtmv.groovy
index 8a91a45c5f8..872d7bf8ec0 100644
--- a/regression-test/suites/mtmv_p0/test_hive_mtmv.groovy
+++ b/regression-test/suites/mtmv_p0/test_hive_mtmv.groovy
@@ -61,14 +61,6 @@ suite("test_hive_mtmv",
"p0,external,hive,external_docker,external_docker_hive")
waitingMTMVTaskFinished(jobName)
order_qt_refresh_one_partition "SELECT * FROM ${mvName} order by
id"
-//refresh other partitions
-// current, for hive, auto refresh will not change data
-sql """
-REFRESH MATERIALIZED VIEW ${mvName} AUTO
-"""
-waitingMTMVTaskFinished(jobName)
-order_qt_refresh_other_partition "SELECT * FROM ${mvName} order by
id"
-
//refresh complete
sql """
REFRESH MATERIALIZED VIEW ${mvName} complete
diff --git a/regression-test/suites/mtmv_p0/test_hive_refresh_mtmv.groovy
b/regression-test/suites/mtmv_p0/test_hive_refresh_mtmv.groovy
index 755c6c15b0d..ac19ecc0605 100644
--- a/regression-test/suites/mtmv_p0/test_hive_refresh_mtmv.groovy
+++ b/regression-test/suites/mtmv_p0/test_hive_refresh_mtmv.groovy
@@ -95,7 +95,7 @@ suite("test_hive_refresh_mtmv",
"p0,external,hive,external_docker,external_docke
assertTrue(showPartitionsResult.toString().contains("p_2020"))
sql """
-REFRESH MATERIALIZED VIEW ${mvName} complete
+REFRESH MATERIALIZED VIEW ${mvName} auto
"""
def jobName = getJobName(dbName, mvName);
waitingMTMVTaskFinished(jobName)
@@ -108,7 +108,7 @@ suite("test_hive_refresh_mtmv",
"p0,external,hive,external_docker,external_docke
REFRESH catalog ${catalog_name}
"""
sql """
-REFRESH MATERIALIZED VIEW ${mvName} complete
+REFRESH MATERIALIZED VIEW ${mvName} auto
"""
waitingMTMVTaskFinished(jobName)
order_qt_mtmv_2 "SELECT * FROM ${mvName} order by user_id"
@@ -124,7 +124,7 @@ suite("test_hive_refresh_mtmv",
"p0,external,hive,external_docker,external_docke
REFRESH catalog ${catalog_name}
"""
sql """
-REFRESH MATERIALIZED VIEW ${mvName} complete
+REFRESH MATERIALIZED VIEW ${mvName} auto
"""
waitingMTMVTaskFinished(jobName)
showPartitionsResult = sql """show partitions from ${mvName}"""
@@ -143,7 +143,7 @@ suite("test_hive_refresh_mtmv",
"p0,external,hive,external_docker,external_docke
REFRESH catalog ${catalog_name}
"""
sql """
-REFRESH MATERIALIZED VIEW ${mvName} complete
+REFRESH MATERIALIZED VIEW ${mvName} auto
"""
waitingMTMVTaskFinished(jobName)
showPartitionsResult = sql """show partitions from ${mvName}"""
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 06/17: [opt](routine-load) optimize routine load task allocation algorithm (#34778)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 05605d99a982d70aaf8c07bea673e86089677ef0
Author: HHoflittlefish777 <77738092+hhoflittlefish...@users.noreply.github.com>
AuthorDate: Fri May 17 11:49:22 2024 +0800
[opt](routine-load) optimize routine load task allocation algorithm (#34778)
---
.../doris/load/routineload/RoutineLoadManager.java | 18 --
1 file changed, 12 insertions(+), 6 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadManager.java
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadManager.java
index 7eb7806f515..356262f8c2a 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadManager.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadManager.java
@@ -488,6 +488,7 @@ public class RoutineLoadManager implements Writable {
readLock();
try {
Map beIdToConcurrentTasks =
getBeCurrentTasksNumMap();
+int previousBeIdleTaskNum = 0;
// 1. Find if the given BE id has more than half of available slots
if (previousBeId != -1L && availableBeIds.contains(previousBeId)) {
@@ -495,22 +496,22 @@ public class RoutineLoadManager implements Writable {
Backend previousBackend =
Env.getCurrentSystemInfo().getBackend(previousBeId);
// check previousBackend is not null && load available
if (previousBackend != null &&
previousBackend.isLoadAvailable()) {
-int idleTaskNum = 0;
if (!beIdToMaxConcurrentTasks.containsKey(previousBeId)) {
-idleTaskNum = 0;
+previousBeIdleTaskNum = 0;
} else if
(beIdToConcurrentTasks.containsKey(previousBeId)) {
-idleTaskNum =
beIdToMaxConcurrentTasks.get(previousBeId)
+previousBeIdleTaskNum =
beIdToMaxConcurrentTasks.get(previousBeId)
- beIdToConcurrentTasks.get(previousBeId);
} else {
-idleTaskNum =
beIdToMaxConcurrentTasks.get(previousBeId);
+previousBeIdleTaskNum =
beIdToMaxConcurrentTasks.get(previousBeId);
}
-if (idleTaskNum > (Config.max_routine_load_task_num_per_be
>> 1)) {
+if (previousBeIdleTaskNum ==
Config.max_routine_load_task_num_per_be) {
return previousBeId;
}
}
}
-// 2. The given BE id does not have available slots, find a BE
with min tasks
+// 2. we believe that the benefits of load balance outweigh the
benefits of object pool cache,
+//so we try to find the one with the most idle slots as much
as possible
// 3. The previous BE is not in cluster && is not load available,
find a new BE with min tasks
int idleTaskNum = 0;
long resultBeId = -1L;
@@ -530,6 +531,11 @@ public class RoutineLoadManager implements Writable {
maxIdleSlotNum = Math.max(maxIdleSlotNum, idleTaskNum);
}
}
+// 4. on the basis of selecting the maximum idle slot be,
+//try to reuse the object cache as much as possible
+if (previousBeIdleTaskNum == maxIdleSlotNum) {
+return previousBeId;
+}
return resultBeId;
} finally {
readUnlock();
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 10/17: [Improve](inverted index) improve performance by introducing bulk contains for bitmap in _build_index_result_column (#34831)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e66dd5886015e29ebb65b06a092558defe836bd5
Author: airborne12
AuthorDate: Fri May 17 14:40:08 2024 +0800
[Improve](inverted index) improve performance by introducing bulk contains
for bitmap in _build_index_result_column (#34831)
---
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
index a74fb305795..bb91a14bd48 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
@@ -2510,12 +2510,13 @@ void SegmentIterator::_build_index_result_column(const
uint16_t* sel_rowid_idx,
vectorized::ColumnUInt8::Container& vec_match_pred =
index_result_column->get_data();
vec_match_pred.resize(block->rows());
size_t idx_in_selected = 0;
+roaring::BulkContext bulk_context;
for (uint32_t i = 0; i < _current_batch_rows_read; i++) {
auto rowid = _block_rowids[i];
if (sel_rowid_idx == nullptr ||
(idx_in_selected < select_size && i ==
sel_rowid_idx[idx_in_selected])) {
-if (index_result.contains(rowid)) {
+if (index_result.containsBulk(bulk_context, rowid)) {
vec_match_pred[idx_in_selected] = true;
} else {
vec_match_pred[idx_in_selected] = false;
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 09/17: [fix](prepare) fix datetimev2 return err when binary_row_format (#34662)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 9b5028785d13a588ab4307e626a5ef26b60ad145
Author: xueweizhang
AuthorDate: Fri May 17 14:19:04 2024 +0800
[fix](prepare) fix datetimev2 return err when binary_row_format (#34662)
fix datetimev2 return err when binary_row_format. before pr, Backend return
datetimev2 alwary by to_string.
fix datatimev2 return metadata loss scale.
---
.../exec/schema_scanner/schema_columns_scanner.cpp | 27 +++--
be/src/util/mysql_row_buffer.cpp | 46 --
be/src/util/mysql_row_buffer.h | 4 +-
.../serde/data_type_datetimev2_serde.cpp | 4 +-
.../org/apache/doris/mysql/MysqlSerializer.java| 2 +
.../test_compaction_uniq_keys_row_store.out| 32 +++
.../scalar_types/sql/infomation_schema.out | 2 +-
7 files changed, 71 insertions(+), 46 deletions(-)
diff --git a/be/src/exec/schema_scanner/schema_columns_scanner.cpp
b/be/src/exec/schema_scanner/schema_columns_scanner.cpp
index 56a6c5f256e..763f24b9e53 100644
--- a/be/src/exec/schema_scanner/schema_columns_scanner.cpp
+++ b/be/src/exec/schema_scanner/schema_columns_scanner.cpp
@@ -512,7 +512,9 @@ Status
SchemaColumnsScanner::_fill_block_impl(vectorized::Block* block) {
{
int64_t srcs[columns_num];
for (int i = 0; i < columns_num; ++i) {
-if (_desc_result.columns[i].columnDesc.__isset.columnPrecision) {
+int data_type = _desc_result.columns[i].columnDesc.columnType;
+if (_desc_result.columns[i].columnDesc.__isset.columnPrecision &&
+data_type != TPrimitiveType::DATETIMEV2) {
srcs[i] = _desc_result.columns[i].columnDesc.columnPrecision;
datas[i] = srcs + i;
} else {
@@ -525,7 +527,9 @@ Status
SchemaColumnsScanner::_fill_block_impl(vectorized::Block* block) {
{
int64_t srcs[columns_num];
for (int i = 0; i < columns_num; ++i) {
-if (_desc_result.columns[i].columnDesc.__isset.columnScale) {
+int data_type = _desc_result.columns[i].columnDesc.columnType;
+if (_desc_result.columns[i].columnDesc.__isset.columnScale &&
+data_type != TPrimitiveType::DATETIMEV2) {
srcs[i] = _desc_result.columns[i].columnDesc.columnScale;
datas[i] = srcs + i;
} else {
@@ -535,7 +539,20 @@ Status
SchemaColumnsScanner::_fill_block_impl(vectorized::Block* block) {
RETURN_IF_ERROR(fill_dest_column_for_range(block, 11, datas));
}
// DATETIME_PRECISION
-{ RETURN_IF_ERROR(fill_dest_column_for_range(block, 12, null_datas)); }
+{
+std::vector srcs(columns_num);
+for (int i = 0; i < columns_num; ++i) {
+int data_type = _desc_result.columns[i].columnDesc.columnType;
+if (_desc_result.columns[i].columnDesc.__isset.columnScale &&
+data_type == TPrimitiveType::DATETIMEV2) {
+srcs[i] = _desc_result.columns[i].columnDesc.columnScale;
+datas[i] = srcs.data() + i;
+} else {
+datas[i] = nullptr;
+}
+}
+RETURN_IF_ERROR(fill_dest_column_for_range(block, 12, datas));
+}
// CHARACTER_SET_NAME
{ RETURN_IF_ERROR(fill_dest_column_for_range(block, 13, null_datas)); }
// COLLATION_NAME
@@ -605,7 +622,9 @@ Status
SchemaColumnsScanner::_fill_block_impl(vectorized::Block* block) {
{
int64_t srcs[columns_num];
for (int i = 0; i < columns_num; ++i) {
-if (_desc_result.columns[i].columnDesc.__isset.columnScale) {
+int data_type = _desc_result.columns[i].columnDesc.columnType;
+if (_desc_result.columns[i].columnDesc.__isset.columnScale &&
+data_type != TPrimitiveType::DATETIMEV2) {
srcs[i] = _desc_result.columns[i].columnDesc.columnScale;
datas[i] = srcs + i;
} else {
diff --git a/be/src/util/mysql_row_buffer.cpp b/be/src/util/mysql_row_buffer.cpp
index a15fa37a0f3..3e4aa332cea 100644
--- a/be/src/util/mysql_row_buffer.cpp
+++ b/be/src/util/mysql_row_buffer.cpp
@@ -385,19 +385,25 @@ int MysqlRowBuffer::push_timev2(double
data, int scale) {
template
template
-int MysqlRowBuffer::push_vec_datetime(DateType& data) {
+int MysqlRowBuffer::push_vec_datetime(DateType& data, int
scale) {
if (is_binary_format && !_dynamic_mode) {
-return push_datetime(data);
+return push_datetime(data, scale);
}
char buf[64];
-char* pos = data.to_string(buf);
+char* pos = nullptr;
+if constexpr (std::is_same_v> ||
+ std::is_same_v>) {
+pos = data.to_string(buf, scale);
+} else {
+pos = data.to_string(buf);
+}
retu
(doris) 13/17: [Fix](Json type) correct cast result for json type (#34764)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e3e5f18f26af29edd0ab7363789d33b60fbc9adf
Author: lihangyu <15605149...@163.com>
AuthorDate: Fri May 17 17:16:14 2024 +0800
[Fix](Json type) correct cast result for json type (#34764)
---
be/src/util/jsonb_document.h | 2 +-
be/src/vec/functions/function_cast.h | 79 +-
be/test/vec/function/function_jsonb_test.cpp | 15 ++--
.../data/datatype_p0/json/json_cast.out| 43
.../data/json_p0/test_json_load_and_function.out | 52 +++---
.../test_json_load_unique_key_and_function.out | 12 ++--
.../data/jsonb_p0/test_jsonb_load_and_function.out | 52 +++---
.../test_jsonb_load_unique_key_and_function.out| 12 ++--
.../data/nereids_function_p0/scalar_function/J.out | 52 +++---
.../json_p0/test_json_load_and_function.out| 52 +++---
.../test_json_load_unique_key_and_function.out | 12 ++--
.../jsonb_p0/test_jsonb_load_and_function.out | 52 +++---
.../test_jsonb_load_unique_key_and_function.out| 12 ++--
regression-test/data/variant_p0/load.out | 17 -
.../suites/datatype_p0/json/json_cast.groovy | 34 ++
regression-test/suites/variant_p0/load.groovy | 25 +--
16 files changed, 307 insertions(+), 216 deletions(-)
diff --git a/be/src/util/jsonb_document.h b/be/src/util/jsonb_document.h
index 73b6a44b38a..4d71cc9f6b3 100644
--- a/be/src/util/jsonb_document.h
+++ b/be/src/util/jsonb_document.h
@@ -498,7 +498,7 @@ public:
bool isNull() const { return (type_ == JsonbType::T_Null); }
bool isTrue() const { return (type_ == JsonbType::T_True); }
bool isFalse() const { return (type_ == JsonbType::T_False); }
-bool isInt() const { return isInt8() || isInt16() || isInt32() ||
isInt64(); }
+bool isInt() const { return isInt8() || isInt16() || isInt32() ||
isInt64() || isInt128(); }
bool isInt8() const { return (type_ == JsonbType::T_Int8); }
bool isInt16() const { return (type_ == JsonbType::T_Int16); }
bool isInt32() const { return (type_ == JsonbType::T_Int32); }
diff --git a/be/src/vec/functions/function_cast.h
b/be/src/vec/functions/function_cast.h
index d19cb4d8574..7f3f74217b4 100644
--- a/be/src/vec/functions/function_cast.h
+++ b/be/src/vec/functions/function_cast.h
@@ -89,6 +89,7 @@
#include "vec/data_types/data_type_struct.h"
#include "vec/data_types/data_type_time.h"
#include "vec/data_types/data_type_time_v2.h"
+#include "vec/data_types/serde/data_type_serde.h"
#include "vec/functions/function.h"
#include "vec/functions/function_convert_tz.h"
#include "vec/functions/function_helpers.h"
@@ -762,29 +763,43 @@ struct ConvertImplGenericToJsonb {
auto column_string = ColumnString::create();
JsonbWriter writer;
+ColumnUInt8::MutablePtr col_null_map_to =
ColumnUInt8::create(col_from.size());
+ColumnUInt8::Container* vec_null_map_to = &col_null_map_to->get_data();
+DataTypeSerDe::FormatOptions format_options;
+format_options.converted_from_string = true;
+DataTypeSerDeSPtr from_serde = type.get_serde();
+DataTypeSerDeSPtr to_serde = data_type_to->get_serde();
+auto col_to = data_type_to->create_column();
+
auto tmp_col = ColumnString::create();
+vectorized::DataTypeSerDe::FormatOptions options;
for (size_t i = 0; i < input_rows_count; i++) {
// convert to string
tmp_col->clear();
VectorBufferWriter write_buffer(*tmp_col.get());
-type.to_string(col_from, i, write_buffer);
+Status st =
+from_serde->serialize_column_to_json(col_from, i, i + 1,
write_buffer, options);
+// if serialized failed, will return null
+(*vec_null_map_to)[i] = !st.ok();
+if (!st.ok()) {
+col_to->insert_default();
+continue;
+}
write_buffer.commit();
writer.reset();
auto str_ref = tmp_col->get_data_at(0);
-ReadBuffer read_buffer((char*)(str_ref.data), str_ref.size);
+Slice data((char*)(str_ref.data), str_ref.size);
// first try to parse string
-Status st = data_type_to->from_string(read_buffer,
column_string.get());
+st = to_serde->deserialize_one_cell_from_json(*col_to, data,
format_options);
+// if parsing failed, will return null
+(*vec_null_map_to)[i] = !st.ok();
if (!st.ok()) {
-// write raw string to jsonb
-writer.writeStartString();
-writer.writeString(str_ref.data, str_ref.size);
-writer.writeEndString();
-column_string->insert_data(writer.getOu
(doris) 08/17: [enhancement](regression-test) modify a key type tests (#34717)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 437c1a1ba40fe6a1e545fbd64b3501eff220e30a
Author: kkop
AuthorDate: Fri May 17 13:48:18 2024 +0800
[enhancement](regression-test) modify a key type tests (#34717)
Co-authored-by: cjj2010 <2449402...@qq.com>
---
.../test_unique_schema_key_change_modify.groovy| 4 +-
.../test_unique_schema_key_change_modify1.groovy | 547 +
2 files changed, 549 insertions(+), 2 deletions(-)
diff --git
a/regression-test/suites/schema_change_p0/test_unique_schema_key_change_modify.groovy
b/regression-test/suites/schema_change_p0/test_unique_schema_key_change_modify.groovy
index c13d1468ada..f33e1f24ade 100644
---
a/regression-test/suites/schema_change_p0/test_unique_schema_key_change_modify.groovy
+++
b/regression-test/suites/schema_change_p0/test_unique_schema_key_change_modify.groovy
@@ -16,8 +16,8 @@
// under the License.
suite("test_unique_schema_key_change_modify","p0") {
- def tbName = "test_unique_schema_key_change_modify"
- def tbName2 = "test_unique_schema_key_change_modify_1"
+ def tbName = "test_unique_schema_key_change_modify_1"
+ def tbName2 = "test_unique_schema_key_change_modify_2"
/**
* Test the unique model by modify a value type
*/
diff --git
a/regression-test/suites/schema_change_p0/test_unique_schema_key_change_modify1.groovy
b/regression-test/suites/schema_change_p0/test_unique_schema_key_change_modify1.groovy
new file mode 100644
index 000..5576362a311
--- /dev/null
+++
b/regression-test/suites/schema_change_p0/test_unique_schema_key_change_modify1.groovy
@@ -0,0 +1,547 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_unique_schema_key_change_modify1","p0") {
+ def tbName = "test_unique_schema_key_change_modify_3"
+ def tbName2 = "test_unique_schema_key_change_modify_4"
+ /**
+ * Test the unique model by modify a value type
+ */
+
+ /**
+ * Test the unique model by modify a key type from DATE to other type
+ */
+
+ sql """ DROP TABLE IF EXISTS ${tbName} """
+ def getTableStatusSql = " SHOW ALTER TABLE COLUMN WHERE
IndexName='${tbName}' ORDER BY createtime DESC LIMIT 1 "
+ def initTable = " CREATE TABLE IF NOT EXISTS ${tbName}\n" +
+ " (\n" +
+ " `user_id` LARGEINT NOT NULL COMMENT \"用户id\",\n" +
+ " `username` VARCHAR(50) NOT NULL COMMENT
\"用户昵称\",\n" +
+ " `login_time` DATE COMMENT \"用户登陆时间\",\n" +
+ " `is_teacher` BOOLEAN COMMENT \"是否是老师\",\n" +
+ " `city` VARCHAR(20) COMMENT \"用户所在城市\",\n" +
+ " `age` SMALLINT COMMENT \"用户年龄\",\n" +
+ " `sex` TINYINT COMMENT \"用户性别\",\n" +
+ " `phone` LARGEINT COMMENT \"用户电话\",\n" +
+ " `address` VARCHAR(500) COMMENT \"用户地址\",\n" +
+ " `register_time` DATETIME COMMENT \"用户注册时间\"\n" +
+ " )\n" +
+ " UNIQUE KEY(`user_id`, `username`, `login_time`)\n" +
+ " DISTRIBUTED BY HASH(`user_id`) BUCKETS 1\n" +
+ " PROPERTIES (\n" +
+ " \"replication_allocation\" = \"tag.location.default:
1\",\n" +
+ " \"enable_unique_key_merge_on_write\" = \"true\"\n" +
+ " );"
+
+ def initTableData = "insert into ${tbName} values(123456789, 'Alice',
'2022-01-01', 0, 'Beijing', 25, 0, 13812345678, 'No. 123 Street, Beijing',
'2022-01-01 10:00:00')," +
+ " (234567890, 'Bob', '2022-01-01 12:00:00', 0,
'Shanghai', 30, 1, 13998765432, 'No. 456 Street, Shanghai', '2022-02-02
12:00:00')," +
+ " (345678901, 'Carol', '2022-01-01 12:00:00', 1,
'Guangzhou', 28, 0, 13724681357, 'No. 789 Street, Guangzhou', '2022-03-03
14:00:00')," +
+ " (456789012, 'Dave', '2022-01-01 12:00:00', 0,
'Shenzhen', 35, 1, 136808
(doris) 15/17: [fix](planner) fix unrequired slot bug when join node introduced by #25204 (#34923)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit a59f9c3fa158ea24da288efba4022261c4826f9e
Author: xueweizhang
AuthorDate: Fri May 17 18:07:31 2024 +0800
[fix](planner) fix unrequired slot bug when join node introduced by #25204
(#34923)
before fix, join node will retain some slots, which are not materialized
and unrequired.
join node need remove these slots and not make them be output slots.
Signed-off-by: nextdreamblue
---
.../org/apache/doris/planner/JoinNodeBase.java | 6 ++
.../test_inlineview_with_project.out | 9 ++-
.../test_inlineview_with_project.groovy| 84 ++
3 files changed, 96 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/JoinNodeBase.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/JoinNodeBase.java
index fbe12b3d6a0..91a3c26e770 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/JoinNodeBase.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/JoinNodeBase.java
@@ -171,6 +171,9 @@ public abstract class JoinNodeBase extends PlanNode {
boolean needSetToNullable =
getChild(0) instanceof JoinNodeBase &&
analyzer.isOuterJoined(leftTupleDesc.getId());
for (SlotDescriptor leftSlotDesc : leftTupleDesc.getSlots()) {
+if (!isMaterializedByChild(leftSlotDesc,
getChild(0).getOutputSmap())) {
+continue;
+}
SlotDescriptor outputSlotDesc =
analyzer.getDescTbl().copySlotDescriptor(outputTupleDesc, leftSlotDesc);
if (leftNullable) {
@@ -191,6 +194,9 @@ public abstract class JoinNodeBase extends PlanNode {
boolean needSetToNullable =
getChild(1) instanceof JoinNodeBase &&
analyzer.isOuterJoined(rightTupleDesc.getId());
for (SlotDescriptor rightSlotDesc : rightTupleDesc.getSlots())
{
+if (!isMaterializedByChild(rightSlotDesc,
getChild(1).getOutputSmap())) {
+continue;
+}
SlotDescriptor outputSlotDesc =
analyzer.getDescTbl().copySlotDescriptor(outputTupleDesc, rightSlotDesc);
if (rightNullable) {
diff --git
a/regression-test/data/correctness_p0/test_inlineview_with_project.out
b/regression-test/data/correctness_p0/test_inlineview_with_project.out
index e6ed3ee5706..1f85a6dff31 100644
--- a/regression-test/data/correctness_p0/test_inlineview_with_project.out
+++ b/regression-test/data/correctness_p0/test_inlineview_with_project.out
@@ -14,9 +14,9 @@
3
-- !select5 --
-1
-2
-3
+1.0
+2.0
+3.0
-- !select5 --
1
@@ -29,3 +29,6 @@
-- !select5 --
3
+-- !select6 --
+2020-01-01
+
diff --git
a/regression-test/suites/correctness_p0/test_inlineview_with_project.groovy
b/regression-test/suites/correctness_p0/test_inlineview_with_project.groovy
index f13c45a5328..e769aa8c8ea 100644
--- a/regression-test/suites/correctness_p0/test_inlineview_with_project.groovy
+++ b/regression-test/suites/correctness_p0/test_inlineview_with_project.groovy
@@ -665,4 +665,88 @@ suite("test_inlineview_with_project") {
sql """DROP TABLE IF EXISTS `dr_user_test_t1`;"""
sql """DROP TABLE IF EXISTS `dr_user_test_t2`;"""
+
+sql """
+drop table if exists dws_mf_wms_join_t1;
+"""
+
+sql """
+drop table if exists dws_mf_wms_join_t2;
+"""
+
+
+sql """CREATE TABLE `dws_mf_wms_join_t1` (
+ `ddate` DATE NULL COMMENT '日期字段',
+ `game_id` VARCHAR(65533) NULL,
+ `main_currency_stock` BIGINT NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`ddate`)
+ DISTRIBUTED BY HASH(`ddate`) BUCKETS 10
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );"""
+
+sql """CREATE TABLE `dws_mf_wms_join_t2` (
+ `game_id` VARCHAR(65533) NULL,
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`game_id`)
+ DISTRIBUTED BY HASH(`game_id`) BUCKETS 10
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );"""
+
+sql """insert into dws_mf_wms_join_t1 values('2020-01-01','12345',100);"""
+sql """insert into dws_mf_wms_join_t2 values('12345');"""
+
+qt_select6 """SELECT
+a1.ddate
+FROM
+(
+SELECT
+aaa.ddate
+FROM
+(
+SELECT
+aa.ddate,
+CONCAT('main', aa.main_currency_stock) AS arr,
+ROW_NUMBER() OVER (
+
(doris) 12/17: [opt](planner)(Nereids) support auto aggregation for random distributed table (#33630)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 81bcb9d4909916dd4bdaee45228838e9e405c69d
Author: Xujian Duan <50550370+darvend...@users.noreply.github.com>
AuthorDate: Fri May 17 16:53:37 2024 +0800
[opt](planner)(Nereids) support auto aggregation for random distributed
table (#33630)
support auto aggregation for querying detail data of random distributed
table:
the same key column will return only one row.
---
.../org/apache/doris/analysis/StmtRewriter.java| 236 +++
.../doris/nereids/jobs/executor/Analyzer.java | 3 +
.../org/apache/doris/nereids/rules/RuleType.java | 5 +-
.../BuildAggForRandomDistributedTable.java | 257 +
.../java/org/apache/doris/qe/StmtExecutor.java | 9 +-
.../aggregate/select_random_distributed_tbl.out| 217 +
.../aggregate/select_random_distributed_tbl.groovy | 134 +++
7 files changed, 857 insertions(+), 4 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java
b/fe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java
index 93823cf398c..8fcd54b4a1d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/StmtRewriter.java
@@ -20,9 +20,16 @@
package org.apache.doris.analysis;
+import org.apache.doris.catalog.AggStateType;
+import org.apache.doris.catalog.AggregateType;
import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.DistributionInfo;
import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.FunctionSet;
+import org.apache.doris.catalog.KeysType;
+import org.apache.doris.catalog.OlapTable;
import org.apache.doris.catalog.ScalarType;
+import org.apache.doris.catalog.TableIf;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.TableAliasGenerator;
@@ -1366,4 +1373,233 @@ public class StmtRewriter {
}
return reAnalyze;
}
+
+/**
+ *
+ * @param column the column of SlotRef
+ * @param selectList new selectList for selectStmt
+ * @param groupByExprs group by Exprs for selectStmt
+ * @return true if ref can be rewritten
+ */
+private static boolean rewriteSelectList(Column column, SelectList
selectList, ArrayList groupByExprs) {
+SlotRef slot = new SlotRef(null, column.getName());
+if (column.isKey()) {
+selectList.addItem(new SelectListItem(slot, column.getName()));
+groupByExprs.add(slot);
+return true;
+} else if (column.isAggregated()) {
+FunctionCallExpr func = generateAggFunction(slot, column);
+if (func != null) {
+selectList.addItem(new SelectListItem(func, column.getName()));
+return true;
+}
+}
+return false;
+}
+
+/**
+ * rewrite stmt for querying random distributed table, construct an
aggregation node for pre-agg
+ * * CREATE TABLE `tbl` (
+ * `k1` BIGINT NULL DEFAULT "10",
+ * `k3` SMALLINT NULL,
+ * `a` BIGINT SUM NULL DEFAULT "0"
+ * ) ENGINE=OLAP
+ * AGGREGATE KEY(`k1`, `k2`)
+ * DISTRIBUTED BY RANDOM BUCKETS 1
+ * PROPERTIES (
+ * "replication_allocation" = "tag.location.default: 1"
+ * )
+ * e.g.,
+ * original: select * from tbl
+ * rewrite: select * from (select k1, k2, sum(pv) from tbl group by k1,
k2) t
+ * do not rewrite if no need two phase agg:
+ * e.g.,
+ * 1. select max(k1) from tbl
+ * 2. select sum(a) from tbl
+ *
+ * @param statementBase stmt to rewrite
+ * @param analyzer the analyzer
+ * @return true if rewritten
+ * @throws UserException
+ */
+public static boolean rewriteForRandomDistribution(StatementBase
statementBase, Analyzer analyzer)
+throws UserException {
+boolean reAnalyze = false;
+if (!(statementBase instanceof SelectStmt)) {
+return false;
+}
+SelectStmt selectStmt = (SelectStmt) statementBase;
+for (int i = 0; i < selectStmt.fromClause.size(); i++) {
+TableRef tableRef = selectStmt.fromClause.get(i);
+// Recursively rewrite subquery
+if (tableRef instanceof InlineViewRef) {
+InlineViewRef viewRef = (InlineViewRef) tableRef;
+if (rewriteForRandomDistribution(viewRef.getQueryStmt(),
viewRef.getAnalyzer())) {
+reAnalyze = true;
+}
+continue;
+}
+TableIf table = tableRef.getTable();
+if (!(table instanceof OlapTable)) {
+continue;
+}
+// only rewrite random distributed AGG_KEY tabl
(doris) 17/17: [fix](hive-ctas) support create hive table with full quolified name (#34984)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 22f85be71266679a6ca4de7657af99cbe7f2ef4e
Author: Mingyu Chen
AuthorDate: Fri May 17 22:52:43 2024 +0800
[fix](hive-ctas) support create hive table with full quolified name (#34984)
Before, when executing `create table hive.db.table as select` to create
table in hive catalog,
if current catalog is not hive catalog, the default engine name will be
filled with `olap`, which is wrong.
This PR will fill the default engine name base on specified catalog.
---
.../trees/plans/commands/info/CreateTableInfo.java | 17 +++
.../external_table_p0/hive/ddl/test_hive_ctas.out | 33 ++
.../hive/ddl/test_hive_ctas.groovy | 14 +
3 files changed, 59 insertions(+), 5 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java
index 585d1da0b10..3d4607517d9 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java
@@ -40,6 +40,7 @@ import org.apache.doris.common.util.AutoBucketUtils;
import org.apache.doris.common.util.InternalDatabaseUtil;
import org.apache.doris.common.util.ParseUtil;
import org.apache.doris.common.util.PropertyAnalyzer;
+import org.apache.doris.datasource.CatalogIf;
import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.datasource.es.EsUtil;
import org.apache.doris.datasource.hive.HMSExternalCatalog;
@@ -548,16 +549,21 @@ public class CreateTableInfo {
}
private void paddingEngineName(String ctlName, ConnectContext ctx) {
+Preconditions.checkArgument(!Strings.isNullOrEmpty(ctlName));
if (Strings.isNullOrEmpty(engineName)) {
-if (InternalCatalog.INTERNAL_CATALOG_NAME.equals(ctlName)) {
+CatalogIf catalog =
Env.getCurrentEnv().getCatalogMgr().getCatalog(ctlName);
+if (catalog == null) {
+throw new AnalysisException("Unknown catalog: " + ctlName);
+}
+
+if (catalog instanceof InternalCatalog) {
engineName = "olap";
-} else if (ctx.getCurrentCatalog() instanceof HMSExternalCatalog) {
+} else if (catalog instanceof HMSExternalCatalog) {
engineName = "hive";
-} else if (ctx.getCurrentCatalog() instanceof
IcebergExternalCatalog) {
+} else if (catalog instanceof IcebergExternalCatalog) {
engineName = "iceberg";
} else {
-// set to olap by default
-engineName = "olap";
+throw new AnalysisException("Current catalog does not support
create table: " + ctlName);
}
}
}
@@ -782,3 +788,4 @@ public class CreateTableInfo {
this.isExternal = isExternal;
}
}
+
diff --git a/regression-test/data/external_table_p0/hive/ddl/test_hive_ctas.out
b/regression-test/data/external_table_p0/hive/ddl/test_hive_ctas.out
index 39d711b2df0..d9fa227a59c 100644
--- a/regression-test/data/external_table_p0/hive/ddl/test_hive_ctas.out
+++ b/regression-test/data/external_table_p0/hive/ddl/test_hive_ctas.out
@@ -39,6 +39,14 @@
\N 11 value_for_pt1
\N 22 value_for_pt11
+-- !qualified_table1 --
+11 value_for_pt1
+22 value_for_pt11
+
+-- !qualified_table2 --
+11 value_for_pt1
+22 value_for_pt11
+
-- !ctas_types_01 --
true 127 32767 2147483647 9223372036854775807 default
22.123453.141592653 9. default default 2023-05-29
2023-05-29T23:19:34
@@ -85,6 +93,14 @@ true 127 32767 2147483647 default 22.12345
3.141592653 9. default
\N 11 value_for_pt1
\N 22 value_for_pt11
+-- !qualified_table1 --
+11 value_for_pt1
+22 value_for_pt11
+
+-- !qualified_table2 --
+11 value_for_pt1
+22 value_for_pt11
+
-- !ctas_types_01 --
true 127 32767 2147483647 9223372036854775807 default
22.123453.141592653 9. default default 2023-05-29
2023-05-29T23:19:34
@@ -131,6 +147,14 @@ true 127 32767 2147483647 default
22.123453.141592653 9. default
\N 11 value_for_pt1
\N 22 value_for_pt11
+-- !qualified_table1 --
+11 value_for_pt1
+22 value_for_pt11
+
+-- !qualified_table2 --
+11 value_for_pt1
+22 value_for_pt11
+
-- !ctas_types_01 --
true 127 32767 2147483647 9223372036854775807 default
22.123453.141592653 9. default default 2023-05-29
(doris) 11/17: [opt](nereids) lazy get expression map when comparing hypergraph (#34753)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit bfd875eae396c10d7c615e9f0071602a189ee809
Author: 谢健
AuthorDate: Fri May 17 15:06:49 2024 +0800
[opt](nereids) lazy get expression map when comparing hypergraph (#34753)
---
.../rules/exploration/mv/HyperGraphComparator.java | 114 +
.../mv/LogicalCompatibilityContext.java| 31 ++
2 files changed, 80 insertions(+), 65 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
index 0c9dc9c33de..d180254c7fe 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/HyperGraphComparator.java
@@ -46,6 +46,7 @@ import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
+import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
@@ -120,30 +121,37 @@ public class HyperGraphComparator {
}
// 3 try to construct a map which can be mapped from edge to edge
-Map queryToView = constructQueryToViewMapWithExpr();
-if (!makeViewJoinCompatible(queryToView)) {
+Map queryToViewJoinEdge =
constructQueryToViewJoinMapWithExpr();
+if (!makeViewJoinCompatible(queryToViewJoinEdge)) {
return ComparisonResult.newInvalidResWithErrorMessage("Join types
are not compatible\n");
}
refreshViewEdges();
// 4 compare them by expression and nodes. Note compare edges after
inferring for nodes
-boolean matchNodes = queryToView.entrySet().stream()
+boolean matchNodes = queryToViewJoinEdge.entrySet().stream()
+.allMatch(e -> compareEdgeWithNode(e.getKey(), e.getValue()));
+if (!matchNodes) {
+return ComparisonResult.newInvalidResWithErrorMessage("Join nodes
are not compatible\n");
+}
+Map queryToViewFilterEdge =
constructQueryToViewFilterMapWithExpr();
+matchNodes = queryToViewFilterEdge.entrySet().stream()
.allMatch(e -> compareEdgeWithNode(e.getKey(), e.getValue()));
if (!matchNodes) {
return ComparisonResult.newInvalidResWithErrorMessage("Join nodes
are not compatible\n");
}
-queryToView.forEach(this::compareEdgeWithExpr);
+queryToViewJoinEdge.forEach(this::compareJoinEdgeWithExpr);
+queryToViewFilterEdge.forEach(this::compareFilterEdgeWithExpr);
// 5 process residual edges
-Sets.difference(getQueryJoinEdgeSet(), queryToView.keySet())
+Sets.difference(getQueryJoinEdgeSet(), queryToViewJoinEdge.keySet())
.forEach(e -> pullUpQueryExprWithEdge.put(e,
e.getExpressions()));
-Sets.difference(getQueryFilterEdgeSet(), queryToView.keySet())
+Sets.difference(getQueryFilterEdgeSet(),
queryToViewFilterEdge.keySet())
.forEach(e -> pullUpQueryExprWithEdge.put(e,
e.getExpressions()));
-Sets.difference(getViewJoinEdgeSet(),
Sets.newHashSet(queryToView.values()))
+Sets.difference(getViewJoinEdgeSet(),
Sets.newHashSet(queryToViewJoinEdge.values()))
.stream()
.filter(e -> !LongBitmap.isOverlap(e.getReferenceNodes(),
eliminateViewNodesMap))
.forEach(e -> pullUpViewExprWithEdge.put(e,
e.getExpressions()));
-Sets.difference(getViewFilterEdgeSet(),
Sets.newHashSet(queryToView.values()))
+Sets.difference(getViewFilterEdgeSet(),
Sets.newHashSet(queryToViewFilterEdge.values()))
.stream()
.filter(e -> !LongBitmap.isOverlap(e.getReferenceNodes(),
eliminateViewNodesMap))
.forEach(e -> pullUpViewExprWithEdge.put(e,
e.getExpressions()));
@@ -238,7 +246,7 @@ public class HyperGraphComparator {
int size = queryExprSetList.size();
for (int i = 0; i < size; i++) {
Set mappingQueryExprSet =
queryExprSetList.get(i).stream()
-.map(e ->
logicalCompatibilityContext.getQueryToViewAllExpressionMapping().get(e))
+.map(logicalCompatibilityContext::getViewNodeExprFromQuery)
.collect(Collectors.toSet());
if (!mappingQueryExprSet.equals(viewExprSetList.get(i))) {
return false;
@@ -350,12 +358,6 @@ public class HyperGraphComparator {
return ImmutableSet.copyOf(queryHyperGraph.getFilterEdges());
}
-private List getQueryEdges() {
-return ImmutableList.builder()
-.addAll(getQueryJoinEdges())
-.addAll(getQueryFilterEdges()).build();
-
(doris) 16/17: [fix](multi-catalog)remove http scheme in oss endpoint (#34907)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 89d5f2e816a75c8f36bf8653e067927eb37d08d8
Author: slothever <18522955+w...@users.noreply.github.com>
AuthorDate: Fri May 17 22:43:59 2024 +0800
[fix](multi-catalog)remove http scheme in oss endpoint (#34907)
remove http scheme in oss endpoint, scheme maybe appear in url
(http://bucket.http//.region.aliyuncs.com) if use http client
---
.../java/org/apache/doris/datasource/property/PropertyConverter.java | 2 +-
.../org/apache/doris/datasource/property/PropertyConverterTest.java | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/PropertyConverter.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/PropertyConverter.java
index bccd3147a9b..e3ba2bdce27 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/PropertyConverter.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/PropertyConverter.java
@@ -493,7 +493,7 @@ public class PropertyConverter {
}
private static String getOssEndpoint(String region, boolean publicAccess) {
-String prefix = "http://oss-";;
+String prefix = "oss-";
String suffix = ".aliyuncs.com";
if (!publicAccess) {
suffix = "-internal" + suffix;
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/PropertyConverterTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/PropertyConverterTest.java
index 90a15db4aaa..4e4b0408641 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/PropertyConverterTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/PropertyConverterTest.java
@@ -336,7 +336,7 @@ public class PropertyConverterTest extends
TestWithFeService {
Map hdProps =
catalog.getCatalogProperty().getHadoopProperties();
Assertions.assertEquals("akk", hdProps.get(OssProperties.ACCESS_KEY));
Assertions.assertEquals("skk", hdProps.get(OssProperties.SECRET_KEY));
-Assertions.assertEquals("http://oss-cn-beijing-internal.aliyuncs.com";,
+Assertions.assertEquals("oss-cn-beijing-internal.aliyuncs.com",
hdProps.get(OssProperties.ENDPOINT));
String queryDlf2 = "create catalog hms_dlf2 properties (\n"
@@ -392,7 +392,7 @@ public class PropertyConverterTest extends
TestWithFeService {
Map hdProps3 =
catalog3.getCatalogProperty().getHadoopProperties();
Assertions.assertEquals("akk", hdProps3.get(OssProperties.ACCESS_KEY));
Assertions.assertEquals("skk", hdProps3.get(OssProperties.SECRET_KEY));
-Assertions.assertEquals("http://oss-cn-beijing-internal.aliyuncs.com";,
hdProps3.get(OssProperties.ENDPOINT));
+Assertions.assertEquals("oss-cn-beijing-internal.aliyuncs.com",
hdProps3.get(OssProperties.ENDPOINT));
}
@Test
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
(doris) 14/17: [enhance](mtmv) MTMV deal partition use name instead of id (#34910)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 435147d449c5b1bbab59c882f0a8ae68adc3c845
Author: zhangdong <493738...@qq.com>
AuthorDate: Fri May 17 17:31:22 2024 +0800
[enhance](mtmv) MTMV deal partition use name instead of id (#34910)
partition id will change when insert overwrite
When the materialized view runs a task, if the base table is in insert
overwrite, the materialized view task may report an error: partition not found
by partitionId
Upgrade compatibility: Hive currently does not support automatic refresh,
so it has no impact
---
.../main/java/org/apache/doris/catalog/MTMV.java | 22 ++--
.../java/org/apache/doris/catalog/OlapTable.java | 27 -
.../doris/datasource/hive/HMSExternalTable.java| 36 ---
.../apache/doris/job/extensions/mtmv/MTMVTask.java | 41
.../doris/mtmv/MTMVMaxTimestampSnapshot.java | 19 ++--
.../org/apache/doris/mtmv/MTMVPartitionUtil.java | 112 ++---
...latedPartitionDescOnePartitionColGenerator.java | 6 +-
.../MTMVRelatedPartitionDescRollUpGenerator.java | 20 ++--
...MTMVRelatedPartitionDescSyncLimitGenerator.java | 6 +-
.../org/apache/doris/mtmv/MTMVRelatedTableIf.java | 6 +-
.../org/apache/doris/mtmv/MTMVRewriteUtil.java | 6 +-
.../doris/mtmv/RelatedPartitionDescResult.java | 12 +--
.../plans/commands/UpdateMvByPartitionCommand.java | 26 ++---
.../apache/doris/mtmv/MTMVPartitionUtilTest.java | 18 ++--
...TMVRelatedPartitionDescRollUpGeneratorTest.java | 28 +++---
.../org/apache/doris/mtmv/MTMVRewriteUtilTest.java | 12 ++-
.../java/org/apache/doris/mtmv/MTMVTaskTest.java | 46 -
17 files changed, 236 insertions(+), 207 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
index fb32741afa3..d95c2388a28 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
@@ -305,12 +305,12 @@ public class MTMV extends OlapTable {
/**
* generateMvPartitionDescs
*
- * @return mvPartitionId ==> mvPartitionKeyDesc
+ * @return mvPartitionName ==> mvPartitionKeyDesc
*/
-public Map generateMvPartitionDescs() {
-Map mtmvItems = getAndCopyPartitionItems();
-Map result = Maps.newHashMap();
-for (Entry entry : mtmvItems.entrySet()) {
+public Map generateMvPartitionDescs() {
+Map mtmvItems = getAndCopyPartitionItems();
+Map result = Maps.newHashMap();
+for (Entry entry : mtmvItems.entrySet()) {
result.put(entry.getKey(), entry.getValue().toPartitionKeyDesc());
}
return result;
@@ -321,19 +321,19 @@ public class MTMV extends OlapTable {
* It is the result of real-time comparison calculation, so there may be
some costs,
* so it should be called with caution
*
- * @return mvPartitionId ==> relationPartitionIds
+ * @return mvPartitionName ==> relationPartitionNames
* @throws AnalysisException
*/
-public Map> calculatePartitionMappings() throws
AnalysisException {
+public Map> calculatePartitionMappings() throws
AnalysisException {
if (mvPartitionInfo.getPartitionType() ==
MTMVPartitionType.SELF_MANAGE) {
return Maps.newHashMap();
}
long start = System.currentTimeMillis();
-Map> res = Maps.newHashMap();
-Map> relatedPartitionDescs =
MTMVPartitionUtil
+Map> res = Maps.newHashMap();
+Map> relatedPartitionDescs =
MTMVPartitionUtil
.generateRelatedPartitionDescs(mvPartitionInfo, mvProperties);
-Map mvPartitionItems = getAndCopyPartitionItems();
-for (Entry entry : mvPartitionItems.entrySet()) {
+Map mvPartitionItems =
getAndCopyPartitionItems();
+for (Entry entry : mvPartitionItems.entrySet())
{
res.put(entry.getKey(),
relatedPartitionDescs.getOrDefault(entry.getValue().toPartitionKeyDesc(),
Sets.newHashSet()));
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index 9cf9be00915..0d3477b3a26 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -95,6 +95,7 @@ import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Map.Entry;
import java.util.Objects;
import java.util.Optional;
import java.util.Set;
@@ -1037,6 +1038,17 @@ public class OlapTable extends Table implements
MTMVRelatedTableIf {
return partition;
}
+public PartitionItem getPartitionItemOrAna
(doris) 05/17: [fix](mtmv)Fix slot desc wrong in query rewrite by materialized view when query is complex (#34904)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit cc11e50200f092006c287684384c2e85f7d94995
Author: seawinde <149132972+seawi...@users.noreply.github.com>
AuthorDate: Fri May 17 11:39:39 2024 +0800
[fix](mtmv)Fix slot desc wrong in query rewrite by materialized view when
query is complex (#34904)
---
.../main/java/org/apache/doris/catalog/MTMV.java | 8 +-
.../mv/AbstractMaterializedViewRule.java | 2 +-
.../mv/InitMaterializationContextHook.java | 5 +-
.../exploration/mv/MaterializationContext.java | 13 +-
.../exploration/mv/MaterializedViewUtils.java | 7 +-
.../trees/plans/visitor/TableCollector.java| 14 +-
.../mv/nested/ddl/customer_create.sql | 15 +
.../mv/nested/ddl/customer_delete.sql | 1 +
.../nereids_rules_p0/mv/nested/ddl/date_create.sql | 24 +
.../nereids_rules_p0/mv/nested/ddl/date_delete.sql | 1 +
.../mv/nested/ddl/lineorder_create.sql | 24 +
.../mv/nested/ddl/lineorder_delete.sql | 1 +
.../mv/nested/ddl/lineorder_flat_create.sql| 45 ++
.../mv/nested/ddl/lineorder_flat_delete.sql| 1 +
.../nereids_rules_p0/mv/nested/ddl/part_create.sql | 16 +
.../nereids_rules_p0/mv/nested/ddl/part_delete.sql | 1 +
.../mv/nested/ddl/supplier_create.sql | 14 +
.../mv/nested/ddl/supplier_delete.sql | 1 +
.../mv/nested/nested_materialized_view.groovy | 750 -
19 files changed, 920 insertions(+), 23 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
index a3050bc3ac6..fb32741afa3 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/MTMV.java
@@ -40,6 +40,7 @@ import org.apache.doris.mtmv.MTMVRefreshSnapshot;
import org.apache.doris.mtmv.MTMVRelation;
import org.apache.doris.mtmv.MTMVStatus;
import org.apache.doris.persist.gson.GsonUtils;
+import org.apache.doris.qe.ConnectContext;
import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
@@ -267,12 +268,15 @@ public class MTMV extends OlapTable {
}
}
-public MTMVCache getOrGenerateCache() throws AnalysisException {
+/**
+ * Called when in query, Should use one connection context in query
+ */
+public MTMVCache getOrGenerateCache(ConnectContext connectionContext)
throws AnalysisException {
if (cache == null) {
writeMvLock();
try {
if (cache == null) {
-this.cache = MTMVCache.from(this,
MTMVPlanUtil.createMTMVContext(this));
+this.cache = MTMVCache.from(this, connectionContext);
}
} finally {
writeMvUnlock();
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
index a88362e6697..2274ad441f8 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewRule.java
@@ -323,9 +323,9 @@ public abstract class AbstractMaterializedViewRule
implements ExplorationRuleFac
continue;
}
recordIfRewritten(queryStructInfo.getOriginalPlan(),
materializationContext);
+rewriteResults.add(rewrittenPlan);
// if rewrite successfully, try to regenerate mv scan because it
maybe used again
materializationContext.tryReGenerateMvScanPlan(cascadesContext);
-rewriteResults.add(rewrittenPlan);
}
return rewriteResults;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java
index 311932fa1b7..d91fa5cb1a9 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java
@@ -63,6 +63,9 @@ public class InitMaterializationContextHook implements
PlannerHook {
}
Plan rewritePlan = cascadesContext.getRewritePlan();
TableCollectorContext collectorContext = new
TableCollectorContext(Sets.newHashSet(), true);
+// Keep use one connection context when in query, if new connect
context,
+// the ConnectionContext.get() will change
+
collectorContext.setConnectContext(cascad
(doris) 07/17: [enhancement](regression-test) modify a key type tests (#34600)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 274c96b12dbacac234c1f2b86492f8383b5c1929
Author: kkop
AuthorDate: Fri May 17 13:44:47 2024 +0800
[enhancement](regression-test) modify a key type tests (#34600)
Co-authored-by: cjj2010 <2449402...@qq.com>
---
.../test_unique_schema_key_change_modify.groovy| 583 -
1 file changed, 580 insertions(+), 3 deletions(-)
diff --git
a/regression-test/suites/schema_change_p0/test_unique_schema_key_change_modify.groovy
b/regression-test/suites/schema_change_p0/test_unique_schema_key_change_modify.groovy
index ace0626b61e..c13d1468ada 100644
---
a/regression-test/suites/schema_change_p0/test_unique_schema_key_change_modify.groovy
+++
b/regression-test/suites/schema_change_p0/test_unique_schema_key_change_modify.groovy
@@ -24,7 +24,7 @@ suite("test_unique_schema_key_change_modify","p0") {
sql """ DROP TABLE IF EXISTS ${tbName} """
def getTableStatusSql = " SHOW ALTER TABLE COLUMN WHERE
IndexName='${tbName}' ORDER BY createtime DESC LIMIT 1 "
- initTable = " CREATE TABLE IF NOT EXISTS ${tbName}\n" +
+ def initTable = " CREATE TABLE IF NOT EXISTS ${tbName}\n" +
" (\n" +
" `user_id` LARGEINT NOT NULL COMMENT \"用户id\",\n" +
" `username` VARCHAR(50) NOT NULL COMMENT
\"用户昵称\",\n" +
@@ -43,7 +43,7 @@ suite("test_unique_schema_key_change_modify","p0") {
" \"enable_unique_key_merge_on_write\" = \"true\"\n" +
" );"
- initTableData = "insert into ${tbName} values(123456789, 'Alice', 0,
'Beijing', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01
10:00:00')," +
+ def initTableData = "insert into ${tbName} values(123456789, 'Alice', 0,
'Beijing', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01
10:00:00')," +
" (234567890, 'Bob', 0, 'Shanghai', 30, 1,
13998765432, 'No. 456 Street, Shanghai', '2022-02-02 12:00:00')," +
" (345678901, 'Carol', 1, 'Guangzhou', 28, 0,
13724681357, 'No. 789 Street, Guangzhou', '2022-03-03 14:00:00')," +
" (456789012, 'Dave', 0, 'Shenzhen', 35, 1,
13680864279, 'No. 987 Street, Shenzhen', '2022-04-04 16:00:00')," +
@@ -52,7 +52,7 @@ suite("test_unique_schema_key_change_modify","p0") {
" (789012345, 'Grace', 0, 'Xian', 29, 0,
133, 'No. 222 Street, Xian', '2022-07-07 22:00:00');"
//TODO Test the unique model by modify a key type from BOOLEAN to TINYINT
- errorMessage="errCode = 2, detailMessage = Can not change BOOLEAN to
TINYINT"
+ def errorMessage="errCode = 2, detailMessage = Can not change BOOLEAN to
TINYINT"
expectException({
sql initTable
sql initTableData
@@ -953,5 +953,582 @@ suite("test_unique_schema_key_change_modify","p0") {
}, insertSql, true, "${tbName}")
}, errorMessage)
+ /**
+ * Test the unique model by modify a key type from FLOAT to other type
+ */
+ sql """ DROP TABLE IF EXISTS ${tbName} """
+ initTable = " CREATE TABLE IF NOT EXISTS ${tbName}\n" +
+ " (\n" +
+ " `user_id` LARGEINT NOT NULL COMMENT \"用户id\",\n" +
+ " `username` VARCHAR(50) NOT NULL COMMENT
\"用户昵称\",\n" +
+ " `score` FLOAT COMMENT \"分数\",\n" +
+ " `city` VARCHAR(20) COMMENT \"用户所在城市\",\n" +
+ " `age` SMALLINT COMMENT \"用户年龄\",\n" +
+ " `sex` TINYINT COMMENT \"用户性别\",\n" +
+ " `phone` LARGEINT COMMENT \"用户电话\",\n" +
+ " `address` VARCHAR(500) COMMENT \"用户地址\",\n" +
+ " `register_time` DATETIME COMMENT \"用户注册时间\"\n" +
+ " )\n" +
+ " UNIQUE KEY(`user_id`, `username`,`score`)\n" +
+ " DISTRIBUTED BY HASH(`user_id`) BUCKETS 1\n" +
+ " PROPERTIES (\n" +
+ " \"replication_allocation\" = \"tag.location.default:
1\",\n" +
+ " \"enable_unique_key_merge_on_write\" = \"true\"\n" +
+ " );"
+
+ initTableData = "insert into ${tbName} values(123456789, 'Alice', 1.8,
'Beijing', 25, 0, 13812345678, 'No. 123 Street, Beijing', '2022-01-01
10:00:00')," +
+ " (234567890, 'Bob', 1.89, 'Shanghai', 30, 1,
13998765432, 'No. 456 Street, Shanghai', '2022-02-02 12:00:00')," +
+ " (345678901, 'Carol', 2.6, 'Guangzhou', 28, 0,
13724681357, 'No. 789 Street, Guangzhou', '2022-03-03 14:00:00')," +
+ " (456789012, 'Dave', 3.9, 'Shenzhen', 35, 1,
13680864279, 'No. 987 Street,
Re: [PR] [fix](agg) memory leak issue in agg operator [doris]
github-actions[bot] commented on PR #35037: URL: https://github.com/apache/doris/pull/35037#issuecomment-2118774262 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](agg) memory leak issue in agg operator [doris]
github-actions[bot] commented on PR #35037: URL: https://github.com/apache/doris/pull/35037#issuecomment-2118774321 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](Variant) fix incorrect use of column index in TabletSchema [doris]
eldenmoon commented on PR #35019: URL: https://github.com/apache/doris/pull/35019#issuecomment-2118780572 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](Variant) fix incorrect use of column index in TabletSchema [doris]
github-actions[bot] commented on PR #35019: URL: https://github.com/apache/doris/pull/35019#issuecomment-2118782137 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](Variant) fix incorrect use of column index in TabletSchema [doris]
doris-robot commented on PR #35019: URL: https://github.com/apache/doris/pull/35019#issuecomment-2118790185 TeamCity be ut coverage result: Function Coverage: 35.70% (9007/25228) Line Coverage: 27.36% (74485/272247) Region Coverage: 26.61% (38528/144796) Branch Coverage: 23.43% (19654/83886) Coverage Report: http://coverage.selectdb-in.cc/coverage/712fe96ec9fa4833df6e516a064bee7eefb55727_712fe96ec9fa4833df6e516a064bee7eefb55727/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Feature][external catalog/lakesoul] support lakesoul catalog [doris]
doris-robot commented on PR #32164: URL: https://github.com/apache/doris/pull/32164#issuecomment-2118798069 TeamCity be ut coverage result: Function Coverage: 35.70% (9008/25233) Line Coverage: 27.36% (74505/272290) Region Coverage: 26.61% (38537/144813) Branch Coverage: 23.44% (19665/83896) Coverage Report: http://coverage.selectdb-in.cc/coverage/5bced06878779e375593b6336d62d1462aa5e84a_5bced06878779e375593b6336d62d1462aa5e84a/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris-website) branch asf-site updated (bdd93e3270d -> 9113d4c29f4)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/doris-website.git discard bdd93e3270d Automated deployment with doris branch @ 7c43d0539daabcb74d4483b9a5472db8228f177f new 9113d4c29f4 Automated deployment with doris branch @ 7c43d0539daabcb74d4483b9a5472db8228f177f This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (bdd93e3270d) \ N -- N -- N refs/heads/asf-site (9113d4c29f4) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/1.2/search-index.json | 2 +- docs/2.0/search-index.json | 2 +- docs/dev/search-index.json | 2 +- search-index.json| 2 +- zh-CN/docs/1.2/search-index.json | 2 +- zh-CN/docs/2.0/search-index.json | 2 +- zh-CN/docs/dev/search-index.json | 2 +- zh-CN/search-index.json | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](fe) exit FE when transfer to (non)master failed [doris]
github-actions[bot] commented on PR #34809: URL: https://github.com/apache/doris/pull/34809#issuecomment-2118818112 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [I] [Feature] Runtime docker image [doris]
linghengqian commented on issue #9426: URL: https://github.com/apache/doris/issues/9426#issuecomment-2118825746 - After completing https://github.com/apache/shardingsphere/issues/31256 , I revisited the current issue but remained uncertain about the initial lack of progress on it. This seems to merely entail building Dockerfiles with multiple different download links for the same Docker image tag, from my perspective, involving three types of Dockerfiles: x64/linux, arm64/linux, and x64noavx2/linux, each separately constructing the FE and BE. Taking the Dockerfile for x64/linux as an example. - For FE, it looks like this. (not verified) Use of https://apache-doris-releases.oss-accelerate.aliyuncs.com/ is from the Doris documentation in https://doris.apache.org/download/ . ```dockerfile FROM openjdk:8u342-jdk ENV JAVA_HOME="/usr/local/openjdk-8/" ENV PATH="/opt/apache-doris/fe/bin:$PATH" apt-get install -y wget wget https://apache-doris-releases.oss-accelerate.aliyuncs.com/apache-doris-2.1.2-bin-x64.tar.gz -O /opt/apache-doris-bin.tar.gz RUN apt-get update && \ apt-get install -y default-mysql-client && \ apt-get clean && \ mkdir /opt/apache-doris && \ cd /opt && \ mv apache-doris-bin/fe /opt/apache-doris/ wget https://raw.githubusercontent.com/apache/doris/2.1.2-rc04/docker/runtime/fe/resource/init_fe.sh -O /opt/apache-doris/fe/bin/init_fe.sh RUN chmod 755 /opt/apache-doris/fe/bin/init_fe.sh ENTRYPOINT ["/opt/apache-doris/fe/bin/init_fe.sh"] ``` - For BE, it looks like this. (not verified) ```dockerfile FROM openjdk:8u342-jdk ENV JAVA_HOME="/usr/local/openjdk-8/" ENV PATH="/opt/apache-doris/be/bin:$PATH" apt-get install -y wget wget https://apache-doris-releases.oss-accelerate.aliyuncs.com/apache-doris-2.1.2-bin-x64.tar.gz -O /opt/apache-doris-bin.tar.gz RUN apt-get update && \ apt-get install -y default-mysql-client && \ apt-get clean && \ mkdir /opt/apache-doris && \ cd /opt && \ mv apache-doris-bin/be /opt/apache-doris/ wget https://github.com/apache/doris/blob/2.1.2-rc04/docker/runtime/be/resource/init_be.sh -O /opt/apache-doris/be/bin/init_be.sh RUN chmod 755 /opt/apache-doris/be/bin/init_be.sh ENTRYPOINT ["/opt/apache-doris/be/bin/init_be.sh"] ``` - If there are no Runtime-level Docker Images available on Docker Hub or ghcr.io, integrating Doris for testing in other projects without pre-building the Docker Images would necessitate multiple downloads of binary files akin to `apache-doris-2.0.3-bin.tar.gz` under testcontainers-java's Java API using ImageFromDockerfile, thereby increasing the overhead during the test setup process. It's hard to say this is an intuitive way to go about it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory [doris]
xinyiZzz commented on code in PR #34985:
URL: https://github.com/apache/doris/pull/34985#discussion_r1605802329
##
be/src/common/daemon.cpp:
##
@@ -247,33 +248,35 @@ void Daemon::memory_gc_thread() {
continue;
}
auto sys_mem_available = doris::MemInfo::sys_mem_available();
-auto proc_mem_no_allocator_cache =
doris::MemInfo::proc_mem_no_allocator_cache();
+auto proc_mem_corrected =
doris::GlobalMemoryArbitrator::proc_mem_corrected();
// GC excess memory for resource groups that not enable overcommit
-auto tg_free_mem = doris::MemInfo::tg_not_enable_overcommit_group_gc();
+auto tg_free_mem = doris::MemInfo::tg_disable_overcommit_group_gc();
sys_mem_available += tg_free_mem;
-proc_mem_no_allocator_cache -= tg_free_mem;
+proc_mem_corrected -= tg_free_mem;
Review Comment:
modify to `process_memory_usage`
##
be/src/runtime/memory/global_memory_arbitrator.h:
##
@@ -0,0 +1,133 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include "util/mem_info.h"
+
+namespace doris {
+
+class GlobalMemoryArbitrator {
+public:
+static inline int64_t proc_mem_no_allocator_cache() {
+return _s_proc_mem_no_allocator_cache.load(std::memory_order_relaxed);
+}
+static inline int64_t proc_mem_corrected() {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory [doris]
xinyiZzz commented on code in PR #34985:
URL: https://github.com/apache/doris/pull/34985#discussion_r1605802370
##
be/src/runtime/memory/thread_mem_tracker_mgr.h:
##
@@ -164,16 +172,35 @@ inline bool
ThreadMemTrackerMgr::push_consumer_tracker(MemTracker* tracker) {
}
_consumer_tracker_stack.push_back(tracker);
tracker->release(_untracked_mem);
+tracker->consume(_reserved_mem); // consume in advance
return true;
}
inline void ThreadMemTrackerMgr::pop_consumer_tracker() {
DCHECK(!_consumer_tracker_stack.empty());
_consumer_tracker_stack.back()->consume(_untracked_mem);
+_consumer_tracker_stack.back()->release(_reserved_mem);
_consumer_tracker_stack.pop_back();
}
inline void ThreadMemTrackerMgr::consume(int64_t size, int
skip_large_memory_check) {
+if (_reserved_mem != 0) {
+if (_reserved_mem >= size) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory [doris]
xinyiZzz commented on code in PR #34985:
URL: https://github.com/apache/doris/pull/34985#discussion_r1605809890
##
be/src/runtime/memory/thread_mem_tracker_mgr.h:
##
@@ -209,14 +236,58 @@ inline void ThreadMemTrackerMgr::flush_untracked_mem() {
_stop_consume = true;
DCHECK(_limiter_tracker_raw);
-old_untracked_mem = _untracked_mem;
-if (_count_scope_mem) _scope_mem += _untracked_mem;
-_limiter_tracker_raw->consume(old_untracked_mem);
-for (auto tracker : _consumer_tracker_stack) {
-tracker->consume(old_untracked_mem);
+_old_untracked_mem = _untracked_mem;
+if (_count_scope_mem) {
+_scope_mem += _untracked_mem;
+}
+_limiter_tracker_raw->consume(_old_untracked_mem);
+for (auto* tracker : _consumer_tracker_stack) {
+tracker->consume(_old_untracked_mem);
}
-_untracked_mem -= old_untracked_mem;
+_untracked_mem -= _old_untracked_mem;
_stop_consume = false;
}
+inline bool ThreadMemTrackerMgr::try_reserve(int64_t size, bool
force_tracker_overcommit) {
Review Comment:
default query/load allows overcommit, if `force_tracker_overcommit` is
false, try_consume will not check the mem_limit of MemTracker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Chore](runtime-filter) adjust some check and error msg on runtime filter [doris]
yiguolei commented on code in PR #35018: URL: https://github.com/apache/doris/pull/35018#discussion_r1605813382 ## be/src/runtime/runtime_filter_mgr.cpp: ## @@ -411,7 +411,8 @@ Status RuntimeFilterMergeControllerEntity::merge(const PMergeFilterRequest* requ RuntimeFilterWrapperHolder holder; RETURN_IF_ERROR(IRuntimeFilter::create_wrapper(¶ms, pool, holder.getHandle())); - RETURN_IF_ERROR(cnt_val->filter->merge_from(holder.getHandle()->get())); +// prevent error ignored +CHECK(cnt_val->filter->merge_from(holder.getHandle()->get())) << "rf merge failed"; Review Comment: why check here？ It will core online. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](agg) memory leak issue in agg operator [doris]
yiguolei merged PR #35037: URL: https://github.com/apache/doris/pull/35037 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(doris) branch master updated: [fix](agg) memory leak issue in agg operator (#35037)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 1525e673294 [fix](agg) memory leak issue in agg operator (#35037)
1525e673294 is described below
commit 1525e6732940f0bc6c8ae07d8c51864c3882cc1f
Author: Jerry Hu
AuthorDate: Sun May 19 00:22:43 2024 +0800
[fix](agg) memory leak issue in agg operator (#35037)
---
be/src/pipeline/dependency.cpp | 6 ++
1 file changed, 6 insertions(+)
diff --git a/be/src/pipeline/dependency.cpp b/be/src/pipeline/dependency.cpp
index 7b25abc5c61..80587af0c8a 100644
--- a/be/src/pipeline/dependency.cpp
+++ b/be/src/pipeline/dependency.cpp
@@ -184,6 +184,12 @@ Status AggSharedState::reset_hash_table() {
}
});
+if (hash_table.has_null_key_data()) {
+auto st = _destroy_agg_status(hash_table.template
get_null_key_data<
+
vectorized::AggregateDataPtr>());
+RETURN_IF_ERROR(st);
+}
+
aggregate_data_container.reset(new
vectorized::AggregateDataContainer(
sizeof(typename HashTableType::key_type),
((total_size_of_aggregate_states +
align_aggregate_states - 1) /
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Chore](runtime-filter) adjust some check and error msg on runtime filter [doris]
BiteThet commented on PR #35018: URL: https://github.com/apache/doris/pull/35018#issuecomment-2118872121 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Chore](runtime-filter) adjust some check and error msg on runtime filter [doris]
github-actions[bot] commented on PR #35018: URL: https://github.com/apache/doris/pull/35018#issuecomment-2118873392 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory [doris]
xinyiZzz commented on code in PR #34985:
URL: https://github.com/apache/doris/pull/34985#discussion_r1605815710
##
be/src/runtime/memory/thread_mem_tracker_mgr.h:
##
@@ -209,14 +236,58 @@ inline void ThreadMemTrackerMgr::flush_untracked_mem() {
_stop_consume = true;
DCHECK(_limiter_tracker_raw);
-old_untracked_mem = _untracked_mem;
-if (_count_scope_mem) _scope_mem += _untracked_mem;
-_limiter_tracker_raw->consume(old_untracked_mem);
-for (auto tracker : _consumer_tracker_stack) {
-tracker->consume(old_untracked_mem);
+_old_untracked_mem = _untracked_mem;
+if (_count_scope_mem) {
+_scope_mem += _untracked_mem;
+}
+_limiter_tracker_raw->consume(_old_untracked_mem);
+for (auto* tracker : _consumer_tracker_stack) {
+tracker->consume(_old_untracked_mem);
}
-_untracked_mem -= old_untracked_mem;
+_untracked_mem -= _old_untracked_mem;
_stop_consume = false;
}
+inline bool ThreadMemTrackerMgr::try_reserve(int64_t size, bool
force_tracker_overcommit) {
Review Comment:
delete `force_tracker_overcommit`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory [doris]
xinyiZzz commented on PR #34985: URL: https://github.com/apache/doris/pull/34985#issuecomment-2118876023 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](memory) Add GlobalMemoryArbitrator and support ReserveMemory [doris]
github-actions[bot] commented on PR #34985: URL: https://github.com/apache/doris/pull/34985#issuecomment-2118877565 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Bug](Variant) fix incorrect use of column index in TabletSchema [doris]
eldenmoon commented on PR #35019: URL: https://github.com/apache/doris/pull/35019#issuecomment-2118880618 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Chore](runtime-filter) adjust some check and error msg on runtime filter [doris]
doris-robot commented on PR #35018: URL: https://github.com/apache/doris/pull/35018#issuecomment-2118882129 TeamCity be ut coverage result: Function Coverage: 35.70% (9006/25228) Line Coverage: 27.36% (74480/272267) Region Coverage: 26.60% (38514/144809) Branch Coverage: 23.43% (19654/83894) Coverage Report: http://coverage.selectdb-in.cc/coverage/5087402e63fb1ddf50ba04d8a81b19fac419d122_5087402e63fb1ddf50ba04d8a81b19fac419d122/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
