This is an automated email from the ASF dual-hosted git repository.
HappenLee pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new ff89892be25 [improvement](be) Prune exact scan keys by tablet bucket
(#68553)
ff89892be25 is described below
commit ff89892be2565b36b2d6fa3b43a8a53ac4e03dc8
Author: HappenLee <[email protected]>
AuthorDate: Mon Oct 12 11:52:05 2026 +0800
[improvement](be) Prune exact scan keys by tablet bucket (#68553)
### What problem does this PR solve?
Issue Number: None
Related PR: #68446
Ordinary IN scans currently send the complete point-key range list to
every selected tablet. FE tablet pruning reduces the tablet set but each
remaining tablet still probes keys belonging to other buckets. For
example, 55 keys across 43 selected tablets produce 2,365 tablet/range
associations.
This change groups final exact point ranges by the table's hash bucket
before creating scanners. FE attaches the full bucket count and ordinal
from the existing tablet map; BE uses the same raw-byte CRC32 as VARCHAR
distribution. Ordinary scanners and both parallel splitting strategies
consume bucket-local ranges, and empty groups skip scanner/segment
loading. Residual predicates and versioned read sources stay on the
normal scan path.
Initial scope: unpartitioned base indexes with one non-null VARCHAR
storage key equal to the sole HASH distribution column. Other schemas,
rollups, special scans, and non-point/coalesced ranges retain the
existing path. `enable_scan_key_bucket_prune` defaults to true and
provides a per-session A/B switch. Optional Thrift metadata preserves
mixed-version behavior. Two Profile counters expose tablet/range
associations before and after grouping.
### Performance validation
Same RELEASE binary, local single FE/BE on a shared 192-logical-CPU
host; 1M MOW rows, 128 buckets, eight insert batches, auto compaction
disabled, 4KB string payload. Each query selects address/chain/payload
for an IN list and `chain = 1`. Parallel scan enabled; short-circuit,
SQL cache and query cache disabled. Each cell has 50 samples; paired
OFF/ON order is randomized and all 1,000 timed query results match.
`doris_cache_cold` clears the dedicated BE's Doris caches before each
sample. OS caches remain uncontrolled; this does not model cold
object-storage reads. Compare OFF/ON within a row, not across cache
modes.
| Cache | IN keys | OFF p50 ms | ON p50 ms | p50 reduction | OFF p95 ms
| ON p95 ms |
|---|---:|---:|---:|---:|---:|---:|
| hot | 4 | 12.306 | 12.327 | -0.2% | 14.861 | 14.358 |
| hot | 9 | 11.956 | 11.738 | 1.8% | 15.550 | 14.778 |
| hot | 30 | 13.324 | 12.665 | 4.9% | 16.209 | 17.543 |
| hot | 55 | 16.221 | 13.691 | 15.6% | 17.982 | 14.849 |
| hot | 100 | 77.752 | 37.955 | 51.2% | 100.982 | 46.330 |
| doris_cache_cold | 4 | 8.263 | 8.257 | 0.1% | 9.886 | 9.912 |
| doris_cache_cold | 9 | 9.583 | 9.320 | 2.7% | 13.308 | 11.846 |
| doris_cache_cold | 30 | 13.207 | 11.947 | 9.5% | 15.255 | 13.696 |
| doris_cache_cold | 55 | 16.720 | 14.012 | 16.2% | 19.495 | 16.214 |
| doris_cache_cold | 100 | 94.023 | 44.950 | 52.2% | 225.355 | 108.331 |
For 55 keys, range associations drop from 2365 to 55 (97.7%); end-to-end
p50 drops about 16%. Separately captured warm profiles show
`GenerateRowRangeByKeysTime` 13.560 → 1.653 ms and cumulative
`ScannerCpuTime` 46.460 → 31.784 ms (single samples). Small lists show
little gain; hot 30-key p95 increased in this run, so this is not a
universal tail-latency improvement.
<details>
<summary>Workload construction and timing controls</summary>
```sql
CREATE TABLE scan_key_bucket_perf (
address VARCHAR(80) NOT NULL, chain INT NOT NULL, payload STRING
) UNIQUE KEY(address) DISTRIBUTED BY HASH(address) BUCKETS 128
PROPERTIES("replication_num"="1", "enable_unique_key_merge_on_write"="true",
"disable_auto_compaction"="true");
```
Insert eight batches of 125,000 rows from `numbers("number"="125000")`.
In batch `b`, let `i = number + b * 125000`; address is
`concat('address', lpad(cast(i as string), 9, '0'))`, chain is `i % 3`,
and payload is `repeat(sha2(cast(i as string), 256), 64)`.
For list size `n`, keys are generated with Python
`random.Random(68446+n).sample(range(1000000), n)` and the same address
formatting. Query all three columns using the generated IN list and
`chain = 1`. Set `max_scan_key_num=1024`,
`max_pushdown_conditions_per_column=1024`, `enable_parallel_scan=true`,
`enable_short_circuit_query=false`, `enable_sql_cache=false`, and
`enable_query_cache=false`. Toggle `enable_scan_key_bucket_prune` per
trial. Hot mode uses five warm-ups per setting; cold mode calls
`/api/clear_cache/all` on the dedicated BE before each sample. Fetch all
rows, measure with `perf_counter_ns`, and compare sorted result
checksums. Profiles are collected separately with both paths warmed.
</details>
### Release note
Improve eligible VARCHAR IN scans by pruning foreign-bucket point ranges
before scanner creation. The `enable_scan_key_bucket_prune` session
variable controls the optimization.
### Check List (For Author)
- Test
- [x] Regression test: generated and replayed
`query_p0/test_scan_key_bucket_prune`; 35 result blocks covering ON/OFF,
parallel ON/OFF, range coalescing, residual predicates, DUP rows, MOW
updates/deletes, and unsupported nullable/composite/partitioned layouts.
Eight setting combinations agree; generated results also match
independent Python references.
- [x] Unit Test: 10 BE tests under ASAN
(`ScanKeyBucketPrunerTest.*:ScannerLateArrivalRfTest.*`) and 14 FE tests
(`OlapScanNodeTest`).
- [x] Manual test: RELEASE BE/FE builds, FE checkstyle, clang-format
16.0.6, build/header hygiene, and the A/B measurements above.
- Behavior changed: Yes, eligible scanners receive bucket-local point
ranges; SQL results and storage formats are unchanged.
- Does this need documentation: No, internal scan optimization; switch
description is included in session-variable metadata.
Static analysis: the repository clang-tidy script was run with commands
expanded from the unity compilation database. The completed run exits
nonzero on an unmatched `NOLINTEND` in `be/src/core/types.h:576`,
confirmed present in the base commit. Other emitted diagnostics also
point to unchanged code. Comparing every emitted diagnostic with the
final diff found no diagnostics on changed lines; this is not reported
as a clean clang-tidy pass.
---
be/src/exec/CMakeLists.txt | 2 +
be/src/exec/operator/olap_scan_operator.cpp | 55 +-
be/src/exec/scan/parallel_scanner_builder.cpp | 22 +-
be/src/exec/scan/parallel_scanner_builder.h | 10 +-
be/src/exec/scan/scan_key_bucket_pruner.cpp | 60 ++
be/src/exec/scan/scan_key_bucket_pruner.h | 42 ++
be/test/exec/operator/olap_scan_operator_test.cpp | 3 +
be/test/exec/scan/scan_key_bucket_pruner_test.cpp | 115 +++
be/test/exec/scan/scanner_late_arrival_rf_test.cpp | 32 +-
.../org/apache/doris/planner/OlapScanNode.java | 30 +
.../java/org/apache/doris/qe/SessionVariable.java | 6 +
.../org/apache/doris/planner/OlapScanNodeTest.java | 88 +++
gensrc/thrift/PlanNodes.thrift | 9 +-
.../data/query_p0/test_scan_key_bucket_prune.out | 798 +++++++++++++++++++++
.../query_p0/test_scan_key_bucket_prune.groovy | 77 ++
15 files changed, 1318 insertions(+), 31 deletions(-)
diff --git a/be/src/exec/CMakeLists.txt b/be/src/exec/CMakeLists.txt
index 7ccbd60b5ea..099224c1576 100644
--- a/be/src/exec/CMakeLists.txt
+++ b/be/src/exec/CMakeLists.txt
@@ -51,6 +51,8 @@ set(EXEC_UNITY_SKIP
${CMAKE_CURRENT_SOURCE_DIR}/operator/operator.cpp
${CMAKE_CURRENT_SOURCE_DIR}/operator/partitioned_aggregation_sink_operator.cpp
${CMAKE_CURRENT_SOURCE_DIR}/operator/scan_operator.cpp
+ # Small standalone helper; preserve the existing scanner/writer unity
batches.
+ ${CMAKE_CURRENT_SOURCE_DIR}/scan/scan_key_bucket_pruner.cpp
${CMAKE_CURRENT_SOURCE_DIR}/sink/writer/vfile_result_writer.cpp)
doris_skip_unity_inclusion(${EXEC_UNITY_SKIP})
set_target_properties(Exec PROPERTIES UNITY_BUILD ${DORIS_UNITY_BUILD}
diff --git a/be/src/exec/operator/olap_scan_operator.cpp
b/be/src/exec/operator/olap_scan_operator.cpp
index c84909ff8ea..d925f827f65 100644
--- a/be/src/exec/operator/olap_scan_operator.cpp
+++ b/be/src/exec/operator/olap_scan_operator.cpp
@@ -34,6 +34,7 @@
#include "exec/runtime_filter/runtime_filter_consumer_helper.h"
#include "exec/scan/olap_scanner.h"
#include "exec/scan/parallel_scanner_builder.h"
+#include "exec/scan/scan_key_bucket_pruner.h"
#include "exprs/function/in.h"
#include "exprs/hybrid_set.h"
#include "exprs/score_runtime.h"
@@ -747,6 +748,37 @@ Status
OlapScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
}
}
+ // Keep the full-scan placeholder in the fallback list. An empty routed
list
+ // means this tablet has no matching keys and must not create a scanner.
+ std::vector<OlapScanRange*> key_ranges;
+ for (auto& range : _cond_ranges) {
+ key_ranges.push_back(range.get());
+ }
+ std::vector<const std::vector<OlapScanRange*>*>
tablet_key_ranges(_tablets.size(), &key_ranges);
+ ScanKeyBucketPruner bucket_pruner;
+ if (p._olap_scan_node.__isset.enable_scan_key_bucket_prune &&
+ p._olap_scan_node.enable_scan_key_bucket_prune) {
+ const int32_t bucket_num = _scan_ranges.front()->bucket_num;
+ DORIS_CHECK_GT(bucket_num, 0);
+ if (bucket_pruner.init(_cond_ranges, bucket_num)) {
+ int64_t routed_ranges = 0;
+ for (size_t i = 0; i < _scan_ranges.size(); ++i) {
+ const auto& range = *_scan_ranges[i];
+ DORIS_CHECK(range.__isset.bucket_seq &&
range.__isset.bucket_num);
+ DORIS_CHECK_EQ(range.bucket_num, bucket_num);
+ DORIS_CHECK_GE(range.bucket_seq, 0);
+ DORIS_CHECK_LT(range.bucket_seq, bucket_num);
+ tablet_key_ranges[i] =
&bucket_pruner.ranges_for_bucket(range.bucket_seq);
+ routed_ranges += tablet_key_ranges[i]->size();
+ }
+ COUNTER_SET(
+ ADD_COUNTER(custom_profile(),
"ScanKeyRangesBeforeBucketPrune", TUnit::UNIT),
+ static_cast<int64_t>(key_ranges.size() * _tablets.size()));
+ COUNTER_SET(ADD_COUNTER(custom_profile(),
"ScanKeyRangesAfterBucketPrune", TUnit::UNIT),
+ routed_ranges);
+ }
+ }
+
bool enable_parallel_scan = state()->enable_parallel_scan();
auto resolve_binlog_scan_type = [](const TPaloScanRange& scan_range) {
if (scan_range.__isset.binlog_scan_type) {
@@ -778,19 +810,9 @@ Status
OlapScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
(_storage_no_merge() || p._olap_scan_node.is_preaggregation)
// binlog<row> need to be read in order
&& !read_row_binlog && !has_tso_predicate) {
- // Filter out the "full scan" placeholder range (has_lower_bound ==
false)
- // so that only ranges with real key bounds are forwarded to the
parallel scanner.
- std::vector<OlapScanRange*> key_ranges;
- for (auto& range : _cond_ranges) {
- if (!range->has_lower_bound) {
- continue;
- }
- key_ranges.emplace_back(range.get());
- }
-
ParallelScannerBuilder scanner_builder(this, _tablets, _read_sources,
_scan_ranges,
- _scanner_profile, key_ranges,
state(), p._limit,
- true,
p._olap_scan_node.is_preaggregation);
+ _scanner_profile,
tablet_key_ranges, state(),
+ p._limit, true,
p._olap_scan_node.is_preaggregation);
int max_scanners_count = state()->parallel_scan_max_scanners_count();
@@ -835,6 +857,9 @@ Status
OlapScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
int scanners_per_tablet = std::max(1, 64 / (int)_scan_ranges.size());
for (size_t scan_range_idx = 0; scan_range_idx < _scan_ranges.size();
scan_range_idx++) {
+ if (tablet_key_ranges[scan_range_idx]->empty()) {
+ continue;
+ }
const auto& palo_scan_range = *_scan_ranges[scan_range_idx];
if (read_row_binlog &&
(palo_scan_range.__isset.start_tso ||
palo_scan_range.__isset.end_tso)) {
@@ -855,7 +880,7 @@ Status
OlapScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
int64_t version = 0;
std::from_chars(palo_scan_range.version.data(),
palo_scan_range.version.data() +
palo_scan_range.version.size(), version);
- std::vector<std::unique_ptr<doris::OlapScanRange>>* ranges =
&_cond_ranges;
+ const auto* ranges = tablet_key_ranges[scan_range_idx];
int size_based_scanners_per_tablet = 1;
if (config::doris_scan_range_max_mb > 0) {
@@ -869,12 +894,12 @@ Status
OlapScanLocalState::_init_scanners(std::list<ScannerSPtr>* scanners) {
int64_t num_ranges = ranges->size();
for (int64_t i = 0; i < num_ranges;) {
std::vector<doris::OlapScanRange*> scanner_ranges;
- scanner_ranges.push_back((*ranges)[i].get());
+ scanner_ranges.push_back((*ranges)[i]);
++i;
for (int64_t j = 1; i < num_ranges && j < ranges_per_scanner &&
(*ranges)[i]->end_include == (*ranges)[i -
1]->end_include;
++j, ++i) {
- scanner_ranges.push_back((*ranges)[i].get());
+ scanner_ranges.push_back((*ranges)[i]);
}
COUNTER_UPDATE(_key_range_counter, scanner_ranges.size());
diff --git a/be/src/exec/scan/parallel_scanner_builder.cpp
b/be/src/exec/scan/parallel_scanner_builder.cpp
index 288c51e1802..e1650f09323 100644
--- a/be/src/exec/scan/parallel_scanner_builder.cpp
+++ b/be/src/exec/scan/parallel_scanner_builder.cpp
@@ -81,6 +81,10 @@ Status
ParallelScannerBuilder::_build_scanners_by_rowid(std::list<ScannerSPtr>&
DCHECK_GE(_rows_per_scanner, _min_rows_per_scanner);
for (size_t tablet_idx = 0; tablet_idx < _tablets.size(); ++tablet_idx) {
+ const auto& key_ranges = *_tablet_key_ranges[tablet_idx];
+ if (key_ranges.empty()) {
+ continue;
+ }
auto&& [tablet, version] = _tablets[tablet_idx];
const auto& scan_range = *_scan_ranges[tablet_idx];
DCHECK(_all_read_sources.contains(tablet->tablet_id()));
@@ -144,7 +148,7 @@ Status
ParallelScannerBuilder::_build_scanners_by_rowid(std::list<ScannerSPtr>&
partitial_read_source.rs_splits.emplace_back(std::move(split));
scanners.emplace_back(_build_scanner(
- tablet, version, _key_ranges, scan_range,
+ tablet, version, key_ranges, scan_range,
{.rs_splits =
std::move(partitial_read_source.rs_splits),
.delete_predicates =
entire_read_source.delete_predicates,
.delete_bitmap =
entire_read_source.delete_bitmap},
@@ -191,7 +195,7 @@ Status
ParallelScannerBuilder::_build_scanners_by_rowid(std::list<ScannerSPtr>&
}
#endif
scanners.emplace_back(
- _build_scanner(tablet, version, _key_ranges, scan_range,
+ _build_scanner(tablet, version, key_ranges, scan_range,
{.rs_splits =
std::move(partitial_read_source.rs_splits),
.delete_predicates =
entire_read_source.delete_predicates,
.delete_bitmap =
entire_read_source.delete_bitmap},
@@ -211,6 +215,10 @@ Status
ParallelScannerBuilder::_build_scanners_by_per_segment(std::list<ScannerS
DCHECK_GE(_rows_per_scanner, _min_rows_per_scanner);
for (size_t tablet_idx = 0; tablet_idx < _tablets.size(); ++tablet_idx) {
+ const auto& key_ranges = *_tablet_key_ranges[tablet_idx];
+ if (key_ranges.empty()) {
+ continue;
+ }
auto&& [tablet, version] = _tablets[tablet_idx];
const auto& scan_range = *_scan_ranges[tablet_idx];
DCHECK(_all_read_sources.contains(tablet->tablet_id()));
@@ -244,7 +252,7 @@ Status
ParallelScannerBuilder::_build_scanners_by_per_segment(std::list<ScannerS
partitial_read_source.rs_splits.emplace_back(std::move(split));
scanners.emplace_back(_build_scanner(
- tablet, version, _key_ranges, scan_range,
+ tablet, version, key_ranges, scan_range,
{.rs_splits =
std::move(partitial_read_source.rs_splits),
.delete_predicates =
entire_read_source.delete_predicates,
.delete_bitmap = entire_read_source.delete_bitmap},
@@ -262,11 +270,14 @@ Status
ParallelScannerBuilder::_build_scanners_by_per_segment(std::list<ScannerS
*/
Status ParallelScannerBuilder::_load() {
_total_rows = 0;
- size_t idx = 0;
bool enable_segment_cache =
_state->query_options().__isset.enable_segment_cache
?
_state->query_options().enable_segment_cache
: true;
- for (auto&& [tablet, version] : _tablets) {
+ for (size_t idx = 0; idx < _tablets.size(); ++idx) {
+ if (_tablet_key_ranges[idx]->empty()) {
+ continue;
+ }
+ auto&& [tablet, version] = _tablets[idx];
const auto tablet_id = tablet->tablet_id();
_all_read_sources[tablet_id] = _read_sources[idx];
const auto& read_source = _all_read_sources[tablet_id];
@@ -288,7 +299,6 @@ Status ParallelScannerBuilder::_load() {
}
_total_rows += rowset->num_rows();
}
- idx++;
}
_rows_per_scanner = _total_rows / _max_scanners_count;
diff --git a/be/src/exec/scan/parallel_scanner_builder.h
b/be/src/exec/scan/parallel_scanner_builder.h
index 60ee95fb267..7a8a704c488 100644
--- a/be/src/exec/scan/parallel_scanner_builder.h
+++ b/be/src/exec/scan/parallel_scanner_builder.h
@@ -45,8 +45,9 @@ public:
std::vector<TabletReadSource>& read_sources,
const std::vector<std::unique_ptr<TPaloScanRange>>&
scan_ranges,
const std::shared_ptr<RuntimeProfile>& profile,
- const std::vector<OlapScanRange*>& key_ranges,
RuntimeState* state,
- int64_t limit, bool is_dup_mow_key, bool
is_preaggregation)
+ const std::vector<const
std::vector<OlapScanRange*>*>& tablet_key_ranges,
+ RuntimeState* state, int64_t limit, bool
is_dup_mow_key,
+ bool is_preaggregation)
: _parent(parent),
_scanner_profile(profile),
_state(state),
@@ -54,10 +55,11 @@ public:
_is_dup_mow_key(is_dup_mow_key),
_is_preaggregation(is_preaggregation),
_tablets(tablets.cbegin(), tablets.cend()),
- _key_ranges(key_ranges.cbegin(), key_ranges.cend()),
+ _tablet_key_ranges(tablet_key_ranges),
_scan_ranges(scan_ranges),
_read_sources(read_sources) {
DORIS_CHECK_EQ(_tablets.size(), scan_ranges.size());
+ DORIS_CHECK_EQ(_tablets.size(), tablet_key_ranges.size());
for (size_t i = 0; i < _tablets.size(); ++i) {
DORIS_CHECK(scan_ranges[i] != nullptr);
DORIS_CHECK_EQ(_tablets[i].tablet->tablet_id(),
scan_ranges[i]->tablet_id);
@@ -119,7 +121,7 @@ private:
// PreAgg OFF: The storage layer must complete pre-aggregation and return
fully aggregated data. (Slow data reading)
bool _is_preaggregation;
std::vector<TabletWithVersion> _tablets;
- std::vector<OlapScanRange*> _key_ranges;
+ const std::vector<const std::vector<OlapScanRange*>*>& _tablet_key_ranges;
const std::vector<std::unique_ptr<TPaloScanRange>>& _scan_ranges;
std::unordered_map<int64_t, TabletReadSource> _all_read_sources;
std::vector<TabletReadSource>& _read_sources;
diff --git a/be/src/exec/scan/scan_key_bucket_pruner.cpp
b/be/src/exec/scan/scan_key_bucket_pruner.cpp
new file mode 100644
index 00000000000..8dc059bea84
--- /dev/null
+++ b/be/src/exec/scan/scan_key_bucket_pruner.cpp
@@ -0,0 +1,60 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/scan/scan_key_bucket_pruner.h"
+
+#include "storage/olap_utils.h"
+#include "util/hash_util.hpp"
+
+namespace doris {
+
+bool ScanKeyBucketPruner::init(const
std::vector<std::unique_ptr<OlapScanRange>>& ranges,
+ int32_t bucket_num) {
+ DORIS_CHECK_GT(bucket_num, 0);
+ _bucket_ranges.clear();
+ for (const auto& range : ranges) {
+ if (!range->has_lower_bound || !range->has_upper_bound ||
!range->begin_include ||
+ !range->end_include || range->begin_scan_range.size() != 1 ||
+ range->end_scan_range.size() != 1) {
+ return false;
+ }
+ const auto& begin = range->begin_scan_range.get_field(0);
+ const auto& end = range->end_scan_range.get_field(0);
+ if (begin.get_type() != TYPE_VARCHAR || end.get_type() != TYPE_VARCHAR
|| begin != end) {
+ return false;
+ }
+ }
+ _bucket_ranges.resize(bucket_num);
+ for (const auto& range : ranges) {
+ const auto& value =
range->begin_scan_range.get_field(0).get<TYPE_VARCHAR>();
+ // Identical to ColumnString::update_crcs_with_value for a single
non-null
+ // VARCHAR distribution column: raw bytes, seed zero, and the full
bucket count.
+ const uint32_t hash =
+ HashUtil::zlib_crc_hash(value.data(),
static_cast<uint32_t>(value.size()), 0);
+ _bucket_ranges[hash % bucket_num].push_back(range.get());
+ }
+ return true;
+}
+
+const std::vector<OlapScanRange*>& ScanKeyBucketPruner::ranges_for_bucket(
+ int32_t bucket_seq) const {
+ DCHECK_GE(bucket_seq, 0);
+ DCHECK_LT(bucket_seq, _bucket_ranges.size());
+ return _bucket_ranges[bucket_seq];
+}
+
+} // namespace doris
diff --git a/be/src/exec/scan/scan_key_bucket_pruner.h
b/be/src/exec/scan/scan_key_bucket_pruner.h
new file mode 100644
index 00000000000..dd1430e0645
--- /dev/null
+++ b/be/src/exec/scan/scan_key_bucket_pruner.h
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+namespace doris {
+
+struct OlapScanRange;
+
+// Groups exact, single-VARCHAR scan keys by the table's HASH distribution
bucket.
+// The caller must establish that this is the table's sole non-null storage and
+// distribution key. Ranges remain owned by the scan local state.
+class ScanKeyBucketPruner {
+public:
+ // Returns false for non-point ranges, including ranges coalesced by
max_scan_key_num.
+ bool init(const std::vector<std::unique_ptr<OlapScanRange>>& ranges,
int32_t bucket_num);
+
+ const std::vector<OlapScanRange*>& ranges_for_bucket(int32_t bucket_seq)
const;
+
+private:
+ std::vector<std::vector<OlapScanRange*>> _bucket_ranges;
+};
+
+} // namespace doris
diff --git a/be/test/exec/operator/olap_scan_operator_test.cpp
b/be/test/exec/operator/olap_scan_operator_test.cpp
index 4cc86c526ee..97aafe54af0 100644
--- a/be/test/exec/operator/olap_scan_operator_test.cpp
+++ b/be/test/exec/operator/olap_scan_operator_test.cpp
@@ -99,6 +99,9 @@ class OlapScanOperatorTsoPruningTest : public
OlapScanOperatorBinlogPushDownTest
protected:
void SetUp() override {
OlapScanOperatorBinlogPushDownTest::SetUp();
+ // Match prepare()'s tablet slots, but leave the tablets null so the
empty-source
+ // tests still verify that TSO pruning finishes without accessing a
tablet.
+ _local_state->_tablets.resize(_local_state->_scan_ranges.size());
_parent->_olap_scan_node.__set_read_row_binlog(true);
_profile = std::make_unique<RuntimeProfile>("TsoPruningTest");
_local_state->_scanner_init_timer = ADD_TIMER(_profile,
"ScannerInitTime");
diff --git a/be/test/exec/scan/scan_key_bucket_pruner_test.cpp
b/be/test/exec/scan/scan_key_bucket_pruner_test.cpp
new file mode 100644
index 00000000000..62cc938473f
--- /dev/null
+++ b/be/test/exec/scan/scan_key_bucket_pruner_test.cpp
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/scan/scan_key_bucket_pruner.h"
+
+#include <gtest/gtest.h>
+
+#include "core/column/column_string.h"
+#include "storage/olap_utils.h"
+
+namespace doris {
+namespace {
+
+std::unique_ptr<OlapScanRange> varchar_point(const std::string& value) {
+ auto range = std::make_unique<OlapScanRange>();
+ range->has_lower_bound = range->has_upper_bound = true;
+
range->begin_scan_range.add_field(Field::create_field<TYPE_VARCHAR>(value));
+ range->end_scan_range.add_field(Field::create_field<TYPE_VARCHAR>(value));
+ return range;
+}
+
+} // namespace
+
+TEST(ScanKeyBucketPrunerTest,
MatchesDistributionHashAndPreservesRangeIdentity) {
+ const std::vector<std::string> values {"", "a", "address000001",
std::string("a\0b", 3),
+ "中文"};
+ // Python zlib.crc32(bytes) % 128, independent of the routing
implementation.
+ const std::vector<uint32_t> expected_buckets {0, 67, 76, 113, 55};
+ auto column = ColumnString::create();
+ std::vector<std::unique_ptr<OlapScanRange>> ranges;
+ for (const auto& value : values) {
+ ranges.push_back(varchar_point(value));
+ column->insert_data(value.data(), value.size());
+ }
+ std::vector<uint32_t> hashes(values.size(), 0);
+ column->update_crcs_with_value(hashes.data(), TYPE_VARCHAR,
+ static_cast<uint32_t>(values.size()), 0,
nullptr);
+ ScanKeyBucketPruner pruner;
+ ASSERT_TRUE(pruner.init(ranges, 128));
+ size_t routed_count = 0;
+ for (int bucket = 0; bucket < 128; ++bucket) {
+ routed_count += pruner.ranges_for_bucket(bucket).size();
+ }
+ EXPECT_EQ(routed_count, ranges.size());
+ for (size_t i = 0; i < ranges.size(); ++i) {
+ EXPECT_EQ(hashes[i] % 128, expected_buckets[i]);
+ const auto& routed = pruner.ranges_for_bucket(expected_buckets[i]);
+ ASSERT_EQ(routed.size(), 1);
+ EXPECT_EQ(routed.front(), ranges[i].get());
+ }
+ EXPECT_TRUE(pruner.ranges_for_bucket(1).empty());
+}
+
+TEST(ScanKeyBucketPrunerTest, KeepsDuplicatesAndBucketLocalOrder) {
+ std::vector<std::unique_ptr<OlapScanRange>> ranges;
+ ranges.push_back(varchar_point("a"));
+ ranges.push_back(varchar_point("a"));
+ ScanKeyBucketPruner pruner;
+ ASSERT_TRUE(pruner.init(ranges, 128));
+ EXPECT_EQ(pruner.ranges_for_bucket(67),
+ (std::vector<OlapScanRange*> {ranges[0].get(),
ranges[1].get()}));
+}
+
+TEST(ScanKeyBucketPrunerTest, RejectsIntervalsAndUnsupportedKeys) {
+ ScanKeyBucketPruner pruner;
+ std::vector<std::unique_ptr<OlapScanRange>> ranges;
+ ranges.push_back(varchar_point("a"));
+ auto check_fallback = [&](std::unique_ptr<OlapScanRange> range) {
+ ASSERT_TRUE(pruner.init(ranges, 128));
+ ranges.push_back(std::move(range));
+ EXPECT_FALSE(pruner.init(ranges, 128));
+ ranges.pop_back();
+ };
+ check_fallback(std::make_unique<OlapScanRange>());
+ auto range = varchar_point("b");
+ range->end_scan_range.get_field(0) =
Field::create_field<TYPE_VARCHAR>("z");
+ check_fallback(std::move(range));
+ range = varchar_point("b");
+ range->begin_include = false;
+ check_fallback(std::move(range));
+ range = varchar_point("b");
+ range->end_include = false;
+ check_fallback(std::move(range));
+ range = varchar_point("b");
+ range->has_upper_bound = false;
+ check_fallback(std::move(range));
+ range = varchar_point("b");
+ range->begin_scan_range.add_null();
+ range->end_scan_range.add_null();
+ check_fallback(std::move(range));
+ range = varchar_point("b");
+ range->begin_scan_range.get_field(0) = Field(TYPE_NULL);
+ range->end_scan_range.get_field(0) = Field(TYPE_NULL);
+ check_fallback(std::move(range));
+ range = varchar_point("b");
+ range->begin_scan_range.get_field(0) = Field::create_field<TYPE_INT>(1);
+ range->end_scan_range.get_field(0) = Field::create_field<TYPE_INT>(1);
+ check_fallback(std::move(range));
+}
+
+} // namespace doris
diff --git a/be/test/exec/scan/scanner_late_arrival_rf_test.cpp
b/be/test/exec/scan/scanner_late_arrival_rf_test.cpp
index 8428a3d6290..a21e7f47bda 100644
--- a/be/test/exec/scan/scanner_late_arrival_rf_test.cpp
+++ b/be/test/exec/scan/scanner_late_arrival_rf_test.cpp
@@ -583,8 +583,10 @@ TEST_F(ScannerLateArrivalRfTest,
parallel_scanner_factory_preserves_bucket_ident
scan_range->__set_bucket_num(bucket_num);
scan_ranges.push_back(std::move(scan_range));
auto profile = std::make_shared<RuntimeProfile>("parallel scanner bucket
identity");
+ const std::vector<OlapScanRange*> key_ranges;
+ const std::vector<const std::vector<OlapScanRange*>*> tablet_key_ranges(1,
&key_ranges);
ParallelScannerBuilder builder(local_state.get(), tablets, read_sources,
scan_ranges, profile,
- {}, state, -1, true, true);
+ tablet_key_ranges, state, -1, true, true);
auto scanner = builder._build_scanner(tablet, 1, {}, *scan_ranges.front(),
TabletReadSource {},
io::FileCacheStatistics {});
@@ -593,6 +595,28 @@ TEST_F(ScannerLateArrivalRfTest,
parallel_scanner_factory_preserves_bucket_ident
EXPECT_TRUE(scanner->is_pruned_by_runtime_filter());
}
+TEST_F(ScannerLateArrivalRfTest,
empty_bucket_keys_skip_loading_and_both_scanner_split_paths) {
+ auto tablet = std::make_shared<FakeTablet>(1, 20);
+ std::vector<TabletWithVersion> tablets {{tablet, 1}};
+ std::vector<TabletReadSource> read_sources(1);
+ std::vector<std::unique_ptr<TPaloScanRange>> scan_ranges;
+ auto scan_range = std::make_unique<TPaloScanRange>();
+ scan_range->__set_tablet_id(20);
+ scan_ranges.push_back(std::move(scan_range));
+ const std::vector<OlapScanRange*> empty_keys;
+ const std::vector<const std::vector<OlapScanRange*>*> tablet_key_ranges(1,
&empty_keys);
+ auto profile = std::make_shared<RuntimeProfile>("empty bucket");
+ for (bool per_segment : {false, true}) {
+ ParallelScannerBuilder builder(nullptr, tablets, read_sources,
scan_ranges, profile,
+ tablet_key_ranges,
_runtime_states[0].get(), -1, true, true);
+ builder.set_scan_parallelism_by_per_segment(per_segment);
+ std::list<ScannerSPtr> scanners;
+ ASSERT_TRUE(builder.build_scanners(scanners).ok());
+ EXPECT_TRUE(scanners.empty());
+ EXPECT_TRUE(builder._all_read_sources.empty());
+ }
+}
+
TEST_F(ScannerLateArrivalRfTest,
high_cardinality_ineligible_parallel_builder_reuses_scan_ranges) {
constexpr size_t scan_range_count = 20'000;
constexpr int64_t first_tablet_id = 100;
@@ -612,8 +636,10 @@ TEST_F(ScannerLateArrivalRfTest,
high_cardinality_ineligible_parallel_builder_re
std::vector<OlapScanRange*> key_ranges;
std::shared_ptr<RuntimeProfile> profile;
- ParallelScannerBuilder builder(nullptr, tablets, read_sources,
scan_ranges, profile, key_ranges,
- nullptr, -1, true, true);
+ const std::vector<const std::vector<OlapScanRange*>*>
tablet_key_ranges(scan_range_count,
+
&key_ranges);
+ ParallelScannerBuilder builder(nullptr, tablets, read_sources,
scan_ranges, profile,
+ tablet_key_ranges, nullptr, -1, true, true);
EXPECT_EQ(&builder._scan_ranges, &scan_ranges);
EXPECT_EQ(builder._scan_ranges.size(), scan_range_count);
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
index fcbd9151002..fce942267a0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
@@ -1528,6 +1528,7 @@ public class OlapScanNode extends ScanNode {
if (hasRfDrivingBucketPruning()) {
setRuntimeFilterBucketPruneParameters();
}
+ setScanKeyBucketPruneParameters(msg.olap_scan_node);
super.toThrift(msg);
}
@@ -1587,6 +1588,35 @@ public class OlapScanNode extends ScanNode {
return false;
}
+ @VisibleForTesting
+ synchronized void setScanKeyBucketPruneParameters(TOlapScanNode node) {
+ ConnectContext context = ConnectContext.get();
+ if (context == null ||
!context.getSessionVariable().enableScanKeyBucketPrune
+ || isPointQuery() || tableSample != null || scanParams != null
+ || olapTable instanceof OlapTableWrapper
+ || selectedIndexId != olapTable.getBaseIndexId()
+ || olapTable.getPartitionInfo().getType() !=
PartitionType.UNPARTITIONED
+ || !(olapTable.getDefaultDistributionInfo() instanceof
HashDistributionInfo)
+ ||
olapTable.getBaseSchema().stream().anyMatch(Column::isClusterKey)) {
+ return;
+ }
+ List<Column> keys = olapTable.getKeyColumnsByIndexId(selectedIndexId);
+ List<Column> distributionColumns =
+ ((HashDistributionInfo)
olapTable.getDefaultDistributionInfo()).getDistributionColumns();
+ if (keys.size() != 1 || distributionColumns.size() != 1
+ || !keys.get(0).equals(distributionColumns.get(0))
+ || !keys.get(0).getType().isVarchar() ||
keys.get(0).isAllowNull()) {
+ return;
+ }
+ setRuntimeFilterBucketPruneParameters();
+ // Missing metadata can deliberately disable attachment (e.g. the
debug point).
+ // Do not attest eligibility unless every range has its full bucket
ordinal/count.
+ if (scanRangeLocations.stream().allMatch(locations ->
+ locations.getScanRange().getPaloScanRange().isSetBucketNum()))
{
+ node.setEnableScanKeyBucketPrune(true);
+ }
+ }
+
@VisibleForTesting
synchronized void setRuntimeFilterBucketPruneParameters() {
if (runtimeFilterBucketPruneParametersSet) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index b3421d02dfd..bc086c7596e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -473,6 +473,8 @@ public class SessionVariable implements Serializable,
Writable {
public static final String ENABLE_RUNTIME_FILTER_BUCKET_PRUNE =
"enable_runtime_filter_bucket_prune";
+ public static final String ENABLE_SCAN_KEY_BUCKET_PRUNE =
"enable_scan_key_bucket_prune";
+
public static final String ENABLE_PRUNE_NESTED_COLUMN =
"enable_prune_nested_column";
static final String SESSION_CONTEXT = "session_context";
@@ -2191,6 +2193,10 @@ public class SessionVariable implements Serializable,
Writable {
@VarAttrDef.VarAttr(name = ENABLE_RUNTIME_FILTER_BUCKET_PRUNE, needForward
= true, fuzzy = true)
public boolean enableRuntimeFilterBucketPrune = true;
+ @VarAttrDef.VarAttr(name = ENABLE_SCAN_KEY_BUCKET_PRUNE, needForward =
true, fuzzy = true,
+ description = "Prune exact VARCHAR scan keys by their HASH
distribution bucket.")
+ public boolean enableScanKeyBucketPrune = true;
+
/**
* The client can pass some special information by setting this session
variable in the format: "k1:v1;k2:v2".
* For example, trace_id can be passed to trace the query request sent by
the user.
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/planner/OlapScanNodeTest.java
b/fe/fe-core/src/test/java/org/apache/doris/planner/OlapScanNodeTest.java
index 44bf4d8215a..e5ea5ab740e 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/planner/OlapScanNodeTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/planner/OlapScanNodeTest.java
@@ -30,6 +30,7 @@ import org.apache.doris.analysis.TupleDescriptor;
import org.apache.doris.analysis.TupleId;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.DiskInfo;
+import org.apache.doris.catalog.HashDistributionInfo;
import org.apache.doris.catalog.LocalReplica;
import org.apache.doris.catalog.LocalTablet;
import org.apache.doris.catalog.MaterializedIndex;
@@ -40,7 +41,9 @@ import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.catalog.RangePartitionInfo;
import org.apache.doris.catalog.RangePartitionItem;
import org.apache.doris.catalog.Replica.ReplicaState;
+import org.apache.doris.catalog.SinglePartitionInfo;
import org.apache.doris.catalog.Tablet;
+import org.apache.doris.catalog.Type;
import org.apache.doris.catalog.info.TableNameInfo;
import org.apache.doris.cloud.catalog.CloudPartition;
import org.apache.doris.common.AnalysisException;
@@ -49,6 +52,8 @@ import org.apache.doris.common.util.DebugPointUtil;
import org.apache.doris.common.util.TimeUtils;
import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.nereids.exceptions.ParseException;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.SessionVariable;
import org.apache.doris.system.Backend;
import org.apache.doris.thrift.TOlapScanNode;
import org.apache.doris.thrift.TPaloScanRange;
@@ -400,6 +405,89 @@ public class OlapScanNodeTest {
}
}
+ @Test
+ public void testScanKeyBucketPruneEligibilityAndFullBucketOrdinals()
throws Exception {
+ ConnectContext context = Mockito.mock(ConnectContext.class,
Mockito.RETURNS_DEEP_STUBS);
+ SessionVariable session = new SessionVariable();
+ Mockito.when(context.getSessionVariable()).thenReturn(session);
+ try (MockedStatic<ConnectContext> contexts =
Mockito.mockStatic(ConnectContext.class)) {
+ contexts.when(ConnectContext::get).thenReturn(context);
+ OlapScanNode scanNode = newBucketPruneScanNode(10L, 11L);
+ OlapTable table = scanNode.getOlapTable();
+ Column key = new Column("address", Type.VARCHAR, true, null,
false, null, "");
+ Mockito.when(table.getBaseIndexId()).thenReturn(1L);
+ Mockito.when(table.getPartitionInfo()).thenReturn(new
SinglePartitionInfo());
+
Mockito.when(table.getBaseSchema()).thenReturn(Lists.newArrayList(key));
+
Mockito.when(table.getKeyColumnsByIndexId(1L)).thenReturn(Lists.newArrayList(key));
+ Mockito.when(table.getDefaultDistributionInfo())
+ .thenReturn(new HashDistributionInfo(128,
Lists.newArrayList(key)));
+ scanNode.setSelectedIndexInfo(1L, true, "");
+ getBucketInfo(scanNode).put(10L, ((long) 128 << Integer.SIZE) |
113L);
+ getBucketInfo(scanNode).put(11L, ((long) 128 << Integer.SIZE) |
67L);
+
+ TOlapScanNode node = new TOlapScanNode();
+ scanNode.setScanKeyBucketPruneParameters(node);
+ Assertions.assertTrue(node.isEnableScanKeyBucketPrune());
+ Assertions.assertEquals(128, scanNode.scanRangeLocations.get(0)
+ .getScanRange().getPaloScanRange().getBucketNum());
+ Assertions.assertEquals(113, scanNode.scanRangeLocations.get(0)
+ .getScanRange().getPaloScanRange().getBucketSeq());
+
+ session.enableScanKeyBucketPrune = false;
+ assertScanKeyBucketPruneDisabled(scanNode);
+ session.enableScanKeyBucketPrune = true;
+ key.setIsAllowNull(true);
+ assertScanKeyBucketPruneDisabled(scanNode);
+ key.setIsAllowNull(false);
+ scanNode.setSelectedIndexInfo(2L, true, "");
+ assertScanKeyBucketPruneDisabled(scanNode);
+ scanNode.setSelectedIndexInfo(1L, true, "");
+
Mockito.when(table.getKeyColumnsByIndexId(1L)).thenReturn(Lists.newArrayList(key,
key));
+ assertScanKeyBucketPruneDisabled(scanNode);
+
Mockito.when(table.getKeyColumnsByIndexId(1L)).thenReturn(Lists.newArrayList(key));
+ Column other = new Column("chain", Type.INT);
+ Mockito.when(table.getDefaultDistributionInfo())
+ .thenReturn(new HashDistributionInfo(128,
Lists.newArrayList(other)));
+ assertScanKeyBucketPruneDisabled(scanNode);
+ Mockito.when(table.getDefaultDistributionInfo())
+ .thenReturn(new HashDistributionInfo(128,
Lists.newArrayList(key, other)));
+ assertScanKeyBucketPruneDisabled(scanNode);
+ Mockito.when(table.getDefaultDistributionInfo())
+ .thenReturn(new HashDistributionInfo(128,
Lists.newArrayList(key)));
+ Mockito.when(table.getPartitionInfo()).thenReturn(new
RangePartitionInfo(Lists.newArrayList(key)));
+ assertScanKeyBucketPruneDisabled(scanNode);
+ }
+ }
+
+ @Test
+ public void testScanKeyBucketPruneRequiresAllTabletMetadata() throws
Exception {
+ ConnectContext context = Mockito.mock(ConnectContext.class,
Mockito.RETURNS_DEEP_STUBS);
+ Mockito.when(context.getSessionVariable()).thenReturn(new
SessionVariable());
+ try (MockedStatic<ConnectContext> contexts =
Mockito.mockStatic(ConnectContext.class)) {
+ contexts.when(ConnectContext::get).thenReturn(context);
+ OlapScanNode scanNode = newBucketPruneScanNode(10L, 11L);
+ OlapTable table = scanNode.getOlapTable();
+ Column key = new Column("address", Type.VARCHAR, true, null,
false, null, "");
+ Mockito.when(table.getBaseIndexId()).thenReturn(1L);
+ Mockito.when(table.getPartitionInfo()).thenReturn(new
SinglePartitionInfo());
+
Mockito.when(table.getBaseSchema()).thenReturn(Lists.newArrayList(key));
+
Mockito.when(table.getKeyColumnsByIndexId(1L)).thenReturn(Lists.newArrayList(key));
+ Mockito.when(table.getDefaultDistributionInfo())
+ .thenReturn(new HashDistributionInfo(128,
Lists.newArrayList(key)));
+ scanNode.setSelectedIndexInfo(1L, true, "");
+ getBucketInfo(scanNode).put(10L, ((long) 128 << Integer.SIZE) |
113L);
+ assertScanKeyBucketPruneDisabled(scanNode);
+ Assertions.assertFalse(scanNode.scanRangeLocations.get(0)
+ .getScanRange().getPaloScanRange().isSetBucketNum());
+ }
+ }
+
+ private void assertScanKeyBucketPruneDisabled(OlapScanNode scanNode) {
+ TOlapScanNode node = new TOlapScanNode();
+ scanNode.setScanKeyBucketPruneParameters(node);
+ Assertions.assertFalse(node.isSetEnableScanKeyBucketPrune());
+ }
+
private OlapScanNode newBucketPruneScanNode(long... tabletIds) {
OlapTable table = Mockito.mock(OlapTable.class);
Mockito.when(table.getName()).thenReturn("rf_bucket_fact");
diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift
index 661bd293463..9699414e9f5 100644
--- a/gensrc/thrift/PlanNodes.thrift
+++ b/gensrc/thrift/PlanNodes.thrift
@@ -102,9 +102,8 @@ struct TPaloScanRange {
10: optional i64 start_tso
11: optional i64 end_tso
12: optional TBinlogScanType binlog_scan_type
- // Bucket metadata for BE-side runtime-filter bucket pruning. These fields
- // are populated only when the scan has an eligible single-column HASH
- // distribution runtime-filter target.
+ // Full distribution bucket ordinal/count for runtime-filter bucket pruning
+ // and exact scan-key bucket pruning. The count is not the selected tablet
count.
13: optional i32 bucket_seq
14: optional i32 bucket_num
}
@@ -1098,6 +1097,10 @@ struct TOlapScanNode {
27: optional list<TPartitionBoundary> partition_boundaries
// Slot ids of extra storage key columns used only to align the scan tuple
with storage schema.
28: optional set<i32> extra_key_column_slot_ids
+ // FE attests: unpartitioned base index, sole non-null VARCHAR storage key
+ // equals the sole HASH distribution key, and all scan ranges carry bucket
metadata.
+ // BE may prune only final inclusive point ranges; absent means disabled.
+ 29: optional bool enable_scan_key_bucket_prune
}
struct TEqJoinCondition {
diff --git a/regression-test/data/query_p0/test_scan_key_bucket_prune.out
b/regression-test/data/query_p0/test_scan_key_bucket_prune.out
new file mode 100644
index 00000000000..0eb76f6af60
--- /dev/null
+++ b/regression-test/data/query_p0/test_scan_key_bucket_prune.out
@@ -0,0 +1,798 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !dup_false_false_1 --
+ 1 empty
+address000001 1 payload1
+address000001 2 duplicate
+address000002 2 payload2
+address000004 1 payload4
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !mow_false_false_1 --
+ 1 empty
+address000001 2 updated
+address000002 1 updated2
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !range_false_false_1 --
+address000003 0 payload3
+address000004 1 payload4
+address000005 2 payload5
+address000006 0 payload6
+address000007 1 payload7
+address000008 2 payload8
+address000009 0 payload9
+
+-- !full_false_false_1 --
+1003 1003
+
+-- !dup_false_false_1024 --
+ 1 empty
+address000001 1 payload1
+address000001 2 duplicate
+address000002 2 payload2
+address000004 1 payload4
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !mow_false_false_1024 --
+ 1 empty
+address000001 2 updated
+address000002 1 updated2
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !range_false_false_1024 --
+address000003 0 payload3
+address000004 1 payload4
+address000005 2 payload5
+address000006 0 payload6
+address000007 1 payload7
+address000008 2 payload8
+address000009 0 payload9
+
+-- !full_false_false_1024 --
+1003 1003
+
+-- !dup_false_true_1 --
+ 1 empty
+address000001 1 payload1
+address000001 2 duplicate
+address000002 2 payload2
+address000004 1 payload4
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !mow_false_true_1 --
+ 1 empty
+address000001 2 updated
+address000002 1 updated2
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !range_false_true_1 --
+address000003 0 payload3
+address000004 1 payload4
+address000005 2 payload5
+address000006 0 payload6
+address000007 1 payload7
+address000008 2 payload8
+address000009 0 payload9
+
+-- !full_false_true_1 --
+1003 1003
+
+-- !dup_false_true_1024 --
+ 1 empty
+address000001 1 payload1
+address000001 2 duplicate
+address000002 2 payload2
+address000004 1 payload4
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !mow_false_true_1024 --
+ 1 empty
+address000001 2 updated
+address000002 1 updated2
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !range_false_true_1024 --
+address000003 0 payload3
+address000004 1 payload4
+address000005 2 payload5
+address000006 0 payload6
+address000007 1 payload7
+address000008 2 payload8
+address000009 0 payload9
+
+-- !full_false_true_1024 --
+1003 1003
+
+-- !dup_true_false_1 --
+ 1 empty
+address000001 1 payload1
+address000001 2 duplicate
+address000002 2 payload2
+address000004 1 payload4
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !mow_true_false_1 --
+ 1 empty
+address000001 2 updated
+address000002 1 updated2
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !range_true_false_1 --
+address000003 0 payload3
+address000004 1 payload4
+address000005 2 payload5
+address000006 0 payload6
+address000007 1 payload7
+address000008 2 payload8
+address000009 0 payload9
+
+-- !full_true_false_1 --
+1003 1003
+
+-- !dup_true_false_1024 --
+ 1 empty
+address000001 1 payload1
+address000001 2 duplicate
+address000002 2 payload2
+address000004 1 payload4
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !mow_true_false_1024 --
+ 1 empty
+address000001 2 updated
+address000002 1 updated2
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !range_true_false_1024 --
+address000003 0 payload3
+address000004 1 payload4
+address000005 2 payload5
+address000006 0 payload6
+address000007 1 payload7
+address000008 2 payload8
+address000009 0 payload9
+
+-- !full_true_false_1024 --
+1003 1003
+
+-- !dup_true_true_1 --
+ 1 empty
+address000001 1 payload1
+address000001 2 duplicate
+address000002 2 payload2
+address000004 1 payload4
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !mow_true_true_1 --
+ 1 empty
+address000001 2 updated
+address000002 1 updated2
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !range_true_true_1 --
+address000003 0 payload3
+address000004 1 payload4
+address000005 2 payload5
+address000006 0 payload6
+address000007 1 payload7
+address000008 2 payload8
+address000009 0 payload9
+
+-- !full_true_true_1 --
+1003 1003
+
+-- !dup_true_true_1024 --
+ 1 empty
+address000001 1 payload1
+address000001 2 duplicate
+address000002 2 payload2
+address000004 1 payload4
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !mow_true_true_1024 --
+ 1 empty
+address000001 2 updated
+address000002 1 updated2
+address000017 2 payload17
+address000034 1 payload34
+address000068 2 payload68
+address000085 1 payload85
+address000119 2 payload119
+address000136 1 payload136
+address000170 2 payload170
+address000187 1 payload187
+address000221 2 payload221
+address000238 1 payload238
+address000272 2 payload272
+address000289 1 payload289
+address000323 2 payload323
+address000340 1 payload340
+address000374 2 payload374
+address000391 1 payload391
+address000425 2 payload425
+address000442 1 payload442
+address000476 2 payload476
+address000493 1 payload493
+address000527 2 payload527
+address000544 1 payload544
+address000578 2 payload578
+address000595 1 payload595
+address000629 2 payload629
+address000646 1 payload646
+address000680 2 payload680
+address000697 1 payload697
+address000731 2 payload731
+address000748 1 payload748
+address000782 2 payload782
+address000799 1 payload799
+address000833 2 payload833
+address000850 1 payload850
+address000884 2 payload884
+address000901 1 payload901
+中文 1 unicode
+
+-- !range_true_true_1024 --
+address000003 0 payload3
+address000004 1 payload4
+address000005 2 payload5
+address000006 0 payload6
+address000007 1 payload7
+address000008 2 payload8
+address000009 0 payload9
+
+-- !full_true_true_1024 --
+1003 1003
+
+-- !nullable --
+\N 0
+a 1
+b 2
+
+-- !composite --
+a 1 1
+b 3 3
+
+-- !partitioned --
+a 1
+c 3
+
diff --git a/regression-test/suites/query_p0/test_scan_key_bucket_prune.groovy
b/regression-test/suites/query_p0/test_scan_key_bucket_prune.groovy
new file mode 100644
index 00000000000..8507bd16436
--- /dev/null
+++ b/regression-test/suites/query_p0/test_scan_key_bucket_prune.groovy
@@ -0,0 +1,77 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_scan_key_bucket_prune") {
+ sql "DROP TABLE IF EXISTS scan_key_bucket_dup"
+ sql """CREATE TABLE scan_key_bucket_dup (
+ address VARCHAR(80) NOT NULL, chain INT NOT NULL, payload STRING
+ ) DUPLICATE KEY(address) DISTRIBUTED BY HASH(address) BUCKETS 128
+ PROPERTIES("replication_num" = "1")"""
+ sql """INSERT INTO scan_key_bucket_dup
+ SELECT concat('address', lpad(cast(number AS STRING), 6, '0')), number
% 3,
+ concat('payload', cast(number AS STRING)) FROM numbers("number"
= "1000")"""
+ sql "INSERT INTO scan_key_bucket_dup VALUES ('', 1, 'empty'), ('中文', 1,
'unicode'), ('address000001', 2, 'duplicate')"
+
+ sql "DROP TABLE IF EXISTS scan_key_bucket_mow"
+ sql """CREATE TABLE scan_key_bucket_mow (
+ address VARCHAR(80) NOT NULL, chain INT NOT NULL, payload STRING
+ ) UNIQUE KEY(address) DISTRIBUTED BY HASH(address) BUCKETS 128
+ PROPERTIES("replication_num" = "1", "enable_unique_key_merge_on_write" =
"true")"""
+ sql "INSERT INTO scan_key_bucket_mow SELECT * FROM scan_key_bucket_dup
WHERE payload != 'duplicate'"
+ sql "INSERT INTO scan_key_bucket_mow VALUES ('address000001', 2,
'updated'), ('address000002', 1, 'updated2')"
+ sql "DELETE FROM scan_key_bucket_mow WHERE address = 'address000004'"
+
+ def keys = (0..<55).collect { "'address${String.format('%06d', it * 17)}'"
}.join(',')
+ keys += ", '', '中文', 'missing', NULL, 'address000001', 'address000002',
'address000004'"
+ [false, true].each { prune ->
+ sql "SET enable_scan_key_bucket_prune = ${prune}"
+ [false, true].each { parallel ->
+ sql "SET enable_parallel_scan = ${parallel}"
+ [1, 1024].each { cap ->
+ // cap=1 coalesces IN keys to a range: routing must fall back.
+ sql "SET max_scan_key_num = ${cap}"
+ def label = "${prune}_${parallel}_${cap}"
+ "order_qt_dup_${label}"("SELECT * FROM scan_key_bucket_dup
WHERE address IN (${keys}) AND chain IN (1, 2)")
+ "order_qt_mow_${label}"("SELECT * FROM scan_key_bucket_mow
WHERE address IN (${keys}) AND chain IN (1, 2)")
+ "order_qt_range_${label}"("SELECT * FROM scan_key_bucket_dup
WHERE address > 'address000002' AND address < 'address000010'")
+ "qt_full_${label}"("SELECT count(*), sum(chain) FROM
scan_key_bucket_dup")
+ }
+ }
+ }
+ sql "SET enable_scan_key_bucket_prune = true"
+ sql "SET max_scan_key_num = 1024"
+ sql "DROP TABLE IF EXISTS scan_key_bucket_nullable"
+ sql """CREATE TABLE scan_key_bucket_nullable (address VARCHAR(80) NULL,
chain INT)
+ DUPLICATE KEY(address) DISTRIBUTED BY HASH(address) BUCKETS 8
+ PROPERTIES("replication_num" = "1")"""
+ sql "INSERT INTO scan_key_bucket_nullable VALUES (NULL, 0), ('a', 1),
('b', 2)"
+ order_qt_nullable "SELECT * FROM scan_key_bucket_nullable WHERE address IN
('a', 'b', NULL) OR address IS NULL"
+
+ sql "DROP TABLE IF EXISTS scan_key_bucket_composite"
+ sql """CREATE TABLE scan_key_bucket_composite (address VARCHAR(80) NOT
NULL, chain INT NOT NULL, v INT)
+ DUPLICATE KEY(address, chain) DISTRIBUTED BY HASH(address, chain)
BUCKETS 8
+ PROPERTIES("replication_num" = "1")"""
+ sql "INSERT INTO scan_key_bucket_composite VALUES ('a', 1, 1), ('a', 2,
2), ('b', 3, 3)"
+ order_qt_composite "SELECT * FROM scan_key_bucket_composite WHERE address
IN ('a', 'b') AND chain IN (1, 3)"
+
+ sql "DROP TABLE IF EXISTS scan_key_bucket_partitioned"
+ sql """CREATE TABLE scan_key_bucket_partitioned (address VARCHAR(80) NOT
NULL, chain INT NOT NULL)
+ DUPLICATE KEY(address) PARTITION BY RANGE(chain) (PARTITION p1 VALUES
LESS THAN ('2'), PARTITION p2 VALUES LESS THAN (MAXVALUE))
+ DISTRIBUTED BY HASH(address) BUCKETS 8 PROPERTIES("replication_num" =
"1")"""
+ sql "INSERT INTO scan_key_bucket_partitioned VALUES ('a', 1), ('b', 2),
('c', 3)"
+ order_qt_partitioned "SELECT * FROM scan_key_bucket_partitioned WHERE
address IN ('a', 'b', 'c') AND chain != 2"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
