This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 9e9bd525c24 branch-4.2: [improvement](zonemap) Support
column-vs-column expression pruning (#67774) (#69659)
9e9bd525c24 is described below
commit 9e9bd525c24b43067f49168c206791cc72996c9e
Author: HappenLee <[email protected]>
AuthorDate: Wed Oct 28 09:23:16 2026 +0800
branch-4.2: [improvement](zonemap) Support column-vs-column expression
pruning (#67774) (#69659)
### What problem does this PR solve?
Issue Number: None
Related PR: #67774
Problem Summary:
Backport #67774 to `branch-4.2`. Comparisons such as `a != b` and `a <
b` can use both columns' zone-map ranges to discard native segments and
Parquet V2 row groups that cannot match. The existing slot-vs-literal
path shares the same six interval rules; dictionary and Bloom-filter
gates remain unchanged.
The backport preserves branch-4.2's `BloomFilterProbe` interfaces
instead of importing the unrelated master metadata-probe/array-contains
refactor. It also retains the original NULL, NaN, string-bound,
unsupported-type and legacy Parquet safeguards, plus the unit tests and
remaining regression coverage.
**Branch-specific adaptation:** master already reads hidden version
columns through a constant reader with substituted zone maps. Branch-4.2
still substitutes `__DORIS_VERSION_COL__` after reading a single-version
rowset, so this backport excludes that column's on-disk placeholder from
both segment- and page-level expression zone-map pruning. The original
enabled/disabled hidden-version regression is retained.
**Deferred VARIANT regression:** omit the VARIANT-path fixture and its
two result blocks until Variant V2 is integrated into branch-4.2. On
this branch, the legacy VARIANT parser returns max-subcolumns 0 while
fuzzy table defaults can differ, so INSERT fails before the
path-comparison checks. The VARIANT exclusion in the production zone-map
gate remains intact. The hidden-version regression is retained.
### Release note
Support column-to-column comparisons in expression zone-map pruning for
native segments and Parquet V2 row groups.
### Check List (For Author)
- Test:
- [ ] Regression test (not executed locally)
- [ ] Unit Test (local configuration blocked)
- [x] Manual test: static backport review, formatting and golden-label
consistency checks only; no runtime execution claimed.
- Passed: `build-support/clang-format.sh` and
`build-support/check-format.sh` with clang-format 16.0.6; `git diff
--check`.
- Carried the original unit tests and remaining regression tests. All 41
remaining `qt_` labels match the golden file. Only the two deferred
VARIANT result blocks were removed from the runner-generated source PR
output; all other result blocks are unchanged.
- Attempted `BUILD_TYPE=ASAN ./build.sh --be -j48`. The cached
third-party package lacks Paimon-Rust headers/archive; stopped the
automatic full dependency rebuild.
- Attempted `BUILD_TYPE=ASAN ./run-be-ut.sh -j48 --run
--filter='ExprZonemapFilterTest.*'`. Configuration stopped because the
checkout lacks the `contrib/openblas` and `contrib/faiss` CMake sources.
No local unit-test or regression runtime pass is claimed; CI validation
is required.
- Behavior changed:
- [x] Yes. Additional predicates can prune; unsupported metadata stays
conservative, and hidden-version placeholders are excluded on this
branch.
- Does this need documentation:
- [x] No. Backport of an existing feature.
No FE/BE protocol, storage-format, configuration, transaction or
data-write changes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases and CI results
- [ ] Confirm document
- [ ] Add branch pick label
---------
Co-authored-by: YangJie <[email protected]>
---
be/src/exprs/expr_zonemap_filter.cpp | 33 ++
be/src/exprs/expr_zonemap_filter.h | 25 ++
be/src/exprs/function/functions_comparison.h | 186 ++++++--
be/src/format/parquet/vparquet_reader.cpp | 41 +-
be/src/format_v2/parquet/parquet_statistics.cpp | 5 +-
.../storage/index/zone_map/zonemap_eval_context.h | 17 +
be/src/storage/segment/segment.cpp | 6 +
be/src/storage/segment/segment_iterator.cpp | 6 +
be/test/exprs/expr_zonemap_filter_test.cpp | 465 +++++++++++++++++++
.../expr_zonemap/test_expr_zonemap_pruning.out | 124 +++++
.../expr_zonemap/test_expr_zonemap_pruning.groovy | 500 +++++++++++++++++++--
11 files changed, 1327 insertions(+), 81 deletions(-)
diff --git a/be/src/exprs/expr_zonemap_filter.cpp
b/be/src/exprs/expr_zonemap_filter.cpp
index 20e12a8c8d7..91b2af4e724 100644
--- a/be/src/exprs/expr_zonemap_filter.cpp
+++ b/be/src/exprs/expr_zonemap_filter.cpp
@@ -265,6 +265,39 @@ std::optional<SlotLiteral> extract_slot_and_literal(const
VExprSPtrs& args) {
return std::nullopt;
}
+std::optional<SlotSlot> extract_slot_and_slot(const VExprSPtrs& args) {
+ if (args.size() != 2) {
+ return std::nullopt;
+ }
+ auto left = std::dynamic_pointer_cast<VSlotRef>(args[0]);
+ auto right = std::dynamic_pointer_cast<VSlotRef>(args[1]);
+ if (left == nullptr || right == nullptr) {
+ return std::nullopt;
+ }
+ return SlotSlot {.left_slot_index = left->column_id(),
+ .left_type = left->data_type(),
+ .right_slot_index = right->column_id(),
+ .right_type = right->data_type()};
+}
+
+bool contains_slot_slot_comparison(const VExprSPtr& expr) {
+ if (expr == nullptr) {
+ return false;
+ }
+ // A comparison node's two operands are its children, so a slot-vs-slot
leaf is a node whose
+ // children are two VSlotRef. extract_slot_and_slot accepts a same-column
pair too, which is the
+ // point: `a < a` has one column id but is still slot-vs-slot.
+ if (extract_slot_and_slot(expr->children()).has_value()) {
+ return true;
+ }
+ for (const auto& child : expr->children()) {
+ if (contains_slot_slot_comparison(child)) {
+ return true;
+ }
+ }
+ return false;
+}
+
std::optional<BloomFilterProbe> extract_bloom_filter_probe(const VExprSPtr&
expr) {
if (expr == nullptr || expr->data_type() == nullptr) {
return std::nullopt;
diff --git a/be/src/exprs/expr_zonemap_filter.h
b/be/src/exprs/expr_zonemap_filter.h
index ee95da8ec46..d247ef57484 100644
--- a/be/src/exprs/expr_zonemap_filter.h
+++ b/be/src/exprs/expr_zonemap_filter.h
@@ -100,6 +100,21 @@ struct SlotLiteral {
bool literal_on_left;
};
+// A comparison whose both operands are plain slot references, e.g. `WHERE a
!= b`. Unlike
+// SlotLiteral there is no flag to flip the operator: with two slots the
argument order already is
+// the expression order, so evaluators use the comparison operator as written.
+struct SlotSlot {
+ // Slot ordinals of the left and right operand. Both are keys used to look
up the corresponding
+ // reader-schema type and zone map from ZoneMapEvalContext, same as
SlotLiteral::slot_index.
+ int left_slot_index;
+ // Type carried by the left slot expression. Kept separately for the same
reason as
+ // SlotLiteral::slot_type: evaluation first has to verify the expression
binding is compatible
+ // with the reader-schema type held by the context.
+ DataTypePtr left_type;
+ int right_slot_index;
+ DataTypePtr right_type;
+};
+
enum class BloomFilterPathKind {
STRUCT_FIELD,
LIST_ELEMENT,
@@ -123,6 +138,16 @@ struct BloomFilterProbe {
std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args);
+// Both operands must be bare VSlotRef. A cast around a slot is deliberately
rejected: the zone map
+// holds raw stored values, so comparing across a cast would need the cast
applied to the bounds.
+std::optional<SlotSlot> extract_slot_and_slot(const VExprSPtrs& args);
+
+// True if the expression tree contains a slot-vs-slot comparison leaf (two
VSlotRef operands),
+// at the root or nested under AND/OR, including two references to the same
column (`a < a`). The v1
+// Parquet reader uses this to exclude slot-vs-slot pruning entirely: counting
distinct columns is
+// not enough because a same-column pair still has one column id.
+bool contains_slot_slot_comparison(const VExprSPtr& expr);
+
std::optional<BloomFilterProbe> extract_bloom_filter_probe(const VExprSPtr&
expr);
std::optional<BloomFilterProbe> extract_bloom_filter_predicate_probe(const
VExprSPtr& expr);
diff --git a/be/src/exprs/function/functions_comparison.h
b/be/src/exprs/function/functions_comparison.h
index a1f681f2e48..affff29d724 100644
--- a/be/src/exprs/function/functions_comparison.h
+++ b/be/src/exprs/function/functions_comparison.h
@@ -49,6 +49,7 @@
#include "exprs/vexpr.h"
#include "runtime/runtime_state.h"
#include "storage/index/index_reader_helper.h"
+#include "storage/olap_define.h"
namespace doris {
#include "common/compile_check_begin.h"
@@ -300,16 +301,43 @@ inline Op symmetric_op(Op op) {
__builtin_unreachable();
}
-inline ZoneMapFilterResult evaluate(const ZoneMapEvalContext& ctx, const
VExprSPtrs& arguments,
- Op op) {
- auto slot_literal = expr_zonemap::extract_slot_and_literal(arguments);
+// Return true when `L <op> R` cannot be TRUE for any row, given that every
non-null value of the
+// left operand lies in [lmin, lmax] and every non-null value of the right
operand lies in
+// [rmin, rmax]. This only compares the four bounds: callers must already have
rejected zone maps
+// whose bounds are unusable and columns that hold no non-null value at all.
+//
+// The slot-vs-literal path passes the literal as the degenerate range
[literal, literal], which
+// reduces every branch below to the plain slot-vs-literal test it replaced.
+inline bool range_vs_range_no_match(const Field& lmin, const Field& lmax,
const Field& rmin,
+ const Field& rmax, Op op) {
+ switch (op) {
+ case Op::EQ:
+ // Disjoint ranges cannot share a value.
+ return lmin > rmax || rmin > lmax;
+ case Op::NE:
+ // Only provable when both sides collapse to the same single value.
+ return lmin == lmax && rmin == rmax && lmin == rmin;
+ case Op::LT:
+ return lmin >= rmax;
+ case Op::LE:
+ return lmin > rmax;
+ case Op::GT:
+ return rmin >= lmax;
+ case Op::GE:
+ return rmin > lmax;
+ }
+ __builtin_unreachable();
+}
- auto slot_type = expr_zonemap::fetch_compatible_slot_type(ctx,
slot_literal->slot_index,
-
slot_literal->slot_type);
+inline ZoneMapFilterResult evaluate_slot_literal(const ZoneMapEvalContext& ctx,
+ const
expr_zonemap::SlotLiteral& slot_literal,
+ Op op) {
+ auto slot_type = expr_zonemap::fetch_compatible_slot_type(ctx,
slot_literal.slot_index,
+
slot_literal.slot_type);
if (slot_type == nullptr) {
return unsupported_zonemap_filter(ctx);
}
- auto zone_map_ptr = ctx.zone_map(slot_literal->slot_index);
+ auto zone_map_ptr = ctx.zone_map(slot_literal.slot_index);
if (zone_map_ptr == nullptr) {
return unsupported_zonemap_filter(ctx);
}
@@ -321,42 +349,95 @@ inline ZoneMapFilterResult evaluate(const
ZoneMapEvalContext& ctx, const VExprSP
return unsupported_zonemap_filter(ctx);
}
- const auto effective_op = slot_literal->literal_on_left ? symmetric_op(op)
: op;
- const auto& literal = slot_literal->literal;
+ const auto effective_op = slot_literal.literal_on_left ? symmetric_op(op)
: op;
+ const auto& literal = slot_literal.literal;
const bool literal_is_nan = literal.is_nan();
const bool hidden_nan_can_match = (effective_op == Op::EQ &&
literal_is_nan) ||
(effective_op == Op::NE &&
!literal_is_nan) ||
(effective_op == Op::GT &&
!literal_is_nan) ||
effective_op == Op::GE;
- if (ctx.floating_nan_count_unknown(slot_literal->slot_index) &&
hidden_nan_can_match) {
+ if (ctx.floating_nan_count_unknown(slot_literal.slot_index) &&
hidden_nan_can_match) {
// Parquet bounds omit NaNs, so only operators that cannot match a
hidden NaN may prune.
return unsupported_zonemap_filter(ctx);
}
- switch (effective_op) {
- case Op::EQ:
- return literal < zone_map.min_value || zone_map.max_value < literal
- ? ZoneMapFilterResult::kNoMatch
- : ZoneMapFilterResult::kMayMatch;
- case Op::NE:
- return zone_map.min_value == literal && zone_map.max_value == literal
- ? ZoneMapFilterResult::kNoMatch
- : ZoneMapFilterResult::kMayMatch;
- case Op::LT:
- return zone_map.min_value >= literal ? ZoneMapFilterResult::kNoMatch
- : ZoneMapFilterResult::kMayMatch;
- case Op::LE:
- return zone_map.min_value > literal ? ZoneMapFilterResult::kNoMatch
- : ZoneMapFilterResult::kMayMatch;
- case Op::GT:
- return zone_map.max_value <= literal ? ZoneMapFilterResult::kNoMatch
- : ZoneMapFilterResult::kMayMatch;
- case Op::GE:
- return zone_map.max_value < literal ? ZoneMapFilterResult::kNoMatch
- : ZoneMapFilterResult::kMayMatch;
+ return range_vs_range_no_match(zone_map.min_value, zone_map.max_value,
literal, literal,
+ effective_op)
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+}
+
+inline ZoneMapFilterResult evaluate_slot_slot(const ZoneMapEvalContext& ctx,
+ const expr_zonemap::SlotSlot&
slot_slot, Op op) {
+ const auto left_type = expr_zonemap::fetch_compatible_slot_type(ctx,
slot_slot.left_slot_index,
+
slot_slot.left_type);
+ const auto right_type = expr_zonemap::fetch_compatible_slot_type(
+ ctx, slot_slot.right_slot_index, slot_slot.right_type);
+ if (left_type == nullptr || right_type == nullptr) {
+ // The context skips a slot entirely when the segment cannot apply
predicates on it.
+ return unsupported_zonemap_filter(ctx);
+ }
+ const auto left_zone_map = ctx.zone_map(slot_slot.left_slot_index);
+ const auto right_zone_map = ctx.zone_map(slot_slot.right_slot_index);
+ if (left_zone_map == nullptr || right_zone_map == nullptr) {
+ // A slot can be present with a data type but no zone map, so this is
a separate check.
+ return unsupported_zonemap_filter(ctx);
+ }
+ // A column holding no non-null value makes the comparison NULL on every
row, which never
+ // satisfies a WHERE conjunct. This must run before the range checks
below: an all-null zone map
+ // leaves min/max default-constructed as TYPE_NULL, which the range checks
would fatal on.
+ if (!left_zone_map->has_not_null || !right_zone_map->has_not_null) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ if (!expr_zonemap::range_stats_usable_for_zonemap(*left_zone_map,
left_type) ||
+ !expr_zonemap::range_stats_usable_for_zonemap(*right_zone_map,
right_type)) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ // A native string zone-map max of exactly MAX_ZONE_MAP_INDEX_SIZE bytes
was bumped upward,
+ // and may also have been truncated (segments written before the carry fix
may hold a bump
+ // that wrapped on 0xff), so a bound of that length is not a reliable
fence for a two-sided
+ // proof. A shorter bound is the exact value; the bump never changes the
length, so this also
+ // rejects those wrapped bounds.
+ // The test only decides whether to trust the bound as a fence. A
Parquet-sourced string bound
+ // shorter than the cap is not necessarily exact, but it is a
spec-conservative min/max, which
+ // is all a range proof needs; do not tighten this to assume exactness
below the cap.
+ // data_types_compatible pairs strings only with strings, so checking one
side's type is enough
+ // to know all four bounds are strings.
+ if (is_string_type(remove_nullable(left_type)->get_primitive_type())) {
+ auto is_untruncated = [](const Field& f) {
+ return f.get<TYPE_STRING>().size() < MAX_ZONE_MAP_INDEX_SIZE;
+ };
+ if (!is_untruncated(left_zone_map->min_value) ||
+ !is_untruncated(left_zone_map->max_value) ||
+ !is_untruncated(right_zone_map->min_value) ||
+ !is_untruncated(right_zone_map->max_value)) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ }
+ // Parquet bounds omit NaN without recording how many were skipped, so a
zone map that looks
+ // like a single point may still hide one, which would flip the NE rule
from false to true. The
+ // slot-vs-literal path can reason per operator because it knows whether
the literal is NaN;
+ // with two slots there is no literal, so bail out for every operator
instead.
+ if (ctx.floating_nan_count_unknown(slot_slot.left_slot_index) ||
+ ctx.floating_nan_count_unknown(slot_slot.right_slot_index)) {
+ return unsupported_zonemap_filter(ctx);
}
+ return range_vs_range_no_match(left_zone_map->min_value,
left_zone_map->max_value,
+ right_zone_map->min_value,
right_zone_map->max_value, op)
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+}
- // keep this to avoid compile failure with g++.
- __builtin_unreachable();
+inline ZoneMapFilterResult evaluate(const ZoneMapEvalContext& ctx, const
VExprSPtrs& arguments,
+ Op op) {
+ if (auto slot_literal = expr_zonemap::extract_slot_and_literal(arguments);
+ slot_literal.has_value()) {
+ return evaluate_slot_literal(ctx, *slot_literal, op);
+ }
+ auto slot_slot = expr_zonemap::extract_slot_and_slot(arguments);
+ // can_evaluate_zonemap_filter accepts exactly the two shapes handled
above, so reaching this
+ // point means the capability gate and this evaluator went out of sync.
+ DORIS_CHECK(slot_slot.has_value());
+ return evaluate_slot_slot(ctx, *slot_slot, op);
}
inline bool can_evaluate(const VExprSPtrs& arguments) {
@@ -384,6 +465,40 @@ inline bool can_evaluate(const VExprSPtrs& arguments) {
return true;
}
+// Accept a comparison whose both operands are slot references. Kept separate
from can_evaluate on
+// purpose: can_evaluate also gates dictionary filtering and
can_evaluate_equality, and both of
+// those dereference extract_slot_and_literal behind a DORIS_CHECK, so
widening it would abort on a
+// slot-vs-slot expression. Only can_evaluate_zonemap_filter ORs this in.
+inline bool can_evaluate_slot_slot(const VExprSPtrs& arguments) {
+ auto slot_slot = expr_zonemap::extract_slot_and_slot(arguments);
+ if (!slot_slot.has_value()) {
+ return false;
+ }
+ DORIS_CHECK(slot_slot->left_type != nullptr);
+ DORIS_CHECK(slot_slot->right_type != nullptr);
+ // Reject types whose slot-slot bounds cannot be compared safely:
+ // - VARBINARY: v1 Parquet produces a TYPE_STRING bound for it, which
+ // range_stats_usable_for_zonemap aborts on instead of falling back.
+ // - VARIANT: a segment types a path slot by its inferred leaf type (a
typed `v.a` is INT), so
+ // fetch_compatible_slot_type's check against the VARIANT expression
type would fail.
+ // - ARRAY/MAP/STRUCT, JSONB, aggregate states: Field orders every pair of
them as equal.
+ // A truncated string bound is rejected later, in evaluate_slot_slot, by
its length.
+ const auto excluded = [](const DataTypePtr& type) {
+ const auto primitive_type =
remove_nullable(type)->get_primitive_type();
+ return is_varbinary(primitive_type) || primitive_type == TYPE_VARIANT
||
+ primitive_type == TYPE_JSONB || is_complex_type(primitive_type)
||
+ is_var_len_object(primitive_type);
+ };
+ if (excluded(slot_slot->left_type) || excluded(slot_slot->right_type)) {
+ return false;
+ }
+ // The two zone maps' Fields are compared directly and Field comparison
throws on mismatched
+ // non-string types, so reject incompatible column pairs here. A pair
differing only by width or
+ // decimal scale never reaches this point anyway, because the optimizer
inserts a cast and a
+ // cast is not a VSlotRef.
+ return expr_zonemap::data_types_compatible(slot_slot->left_type,
slot_slot->right_type);
+}
+
inline bool can_evaluate_equality(const VExprSPtrs& arguments, Op op) {
if (op != Op::EQ || !can_evaluate(arguments)) {
return false;
@@ -657,8 +772,11 @@ public:
}
bool can_evaluate_zonemap_filter(const VExprSPtrs& arguments) const
override {
- return comparison_zonemap_detail::op_from_name(name).has_value() &&
- comparison_zonemap_detail::can_evaluate(arguments);
+ if (!comparison_zonemap_detail::op_from_name(name).has_value()) {
+ return false;
+ }
+ return comparison_zonemap_detail::can_evaluate(arguments) ||
+ comparison_zonemap_detail::can_evaluate_slot_slot(arguments);
}
ZoneMapFilterResult evaluate_dictionary_filter(const
DictionaryEvalContext& ctx,
diff --git a/be/src/format/parquet/vparquet_reader.cpp
b/be/src/format/parquet/vparquet_reader.cpp
index 9ec55437c33..d06e8556f4e 100644
--- a/be/src/format/parquet/vparquet_reader.cpp
+++ b/be/src/format/parquet/vparquet_reader.cpp
@@ -1018,6 +1018,11 @@ bool ParquetReader::_expr_zonemap_page_slot_index(const
VExprContextSPtr& conjun
if (column_ids.size() != 1) {
return false;
}
+ // The v1 reader does not take on slot-vs-slot pruning. A same-column pair
like `a < a` still has
+ // one column id, so exclude any slot-vs-slot leaf explicitly rather than
by column count.
+ if (expr_zonemap::contains_slot_slot_comparison(conjunct->root())) {
+ return false;
+ }
const int slot_index = *column_ids.begin();
if (slot_index < 0 || static_cast<size_t>(slot_index) >=
_tuple_descriptor->slots().size()) {
@@ -1274,6 +1279,11 @@ Status ParquetReader::_process_expr_zonemap_page_filter(
if (!cached_page_index->get_stat_func(&stat, cid) || stat == nullptr
|| !stat->available) {
continue;
}
+ // Same domain requirement as the row-group path above.
+ if (stat->col_schema == nullptr ||
+ !expr_zonemap::data_types_compatible(stat->col_schema->data_type,
slot->type())) {
+ continue;
+ }
RowRanges expr_ranges;
ZoneMapEvalStats page_stats;
for (int64_t page_id = 0; page_id < stat->num_of_pages; ++page_id) {
@@ -1285,7 +1295,7 @@ Status ParquetReader::_process_expr_zonemap_page_filter(
ZoneMapEvalContext ctx;
ZoneMapEvalContext::SlotZoneMap slot_zone_map;
- slot_zone_map.data_type = slot->type();
+ slot_zone_map.set_data_type_from_parquet(slot->type());
segment_v2::ZoneMap zone_map;
zone_map.has_null = stat->has_null[page_id];
zone_map.has_not_null = !stat->is_all_null[page_id];
@@ -1520,11 +1530,24 @@ Status
ParquetReader::_process_expr_zonemap_filter(const tparquet::RowGroup& row
return Status::OK();
}
+ // The v1 Parquet reader is being removed, so it does not take on the new
slot-vs-slot pruning.
+ // Restrict this path to single-column conjuncts that carry no
slot-vs-slot leaf: counting
+ // distinct columns alone is not enough, because a same-column pair like
`a < a` still has one
+ // column id. A two-slot shape falls back to no pruning here. Native and
v2 keep it.
std::set<int> column_ids;
+ VExprContextSPtrs single_slot_conjuncts;
for (const auto& conjunct : all_conjuncts) {
- if (conjunct->root() != nullptr &&
conjunct->root()->can_evaluate_zonemap_filter()) {
- conjunct->root()->collect_slot_column_ids(column_ids);
+ if (conjunct->root() == nullptr ||
!conjunct->root()->can_evaluate_zonemap_filter()) {
+ continue;
}
+ std::set<int> conjunct_column_ids;
+ conjunct->root()->collect_slot_column_ids(conjunct_column_ids);
+ if (conjunct_column_ids.size() != 1 ||
+ expr_zonemap::contains_slot_slot_comparison(conjunct->root())) {
+ continue;
+ }
+ single_slot_conjuncts.emplace_back(conjunct);
+ column_ids.insert(*conjunct_column_ids.begin());
}
if (column_ids.empty()) {
return Status::OK();
@@ -1537,7 +1560,7 @@ Status ParquetReader::_process_expr_zonemap_filter(const
tparquet::RowGroup& row
}
auto* slot = _tuple_descriptor->slots()[cid];
ZoneMapEvalContext::SlotZoneMap slot_zone_map;
- slot_zone_map.data_type = slot->type();
+ slot_zone_map.set_data_type_from_parquet(slot->type());
if (!_exists_in_file(slot->col_name()) || !_type_matches(cid)) {
ctx.slots.emplace(cid, std::move(slot_zone_map));
continue;
@@ -1545,6 +1568,14 @@ Status ParquetReader::_process_expr_zonemap_filter(const
tparquet::RowGroup& row
const auto& file_col_name =
_table_info_node_ptr->children_file_column_name(slot->col_name());
const FieldSchema* col_schema =
_file_metadata->schema().get_column(file_col_name);
+ // parse_min_max_value decodes the bounds in the file's own logical
type, while _type_matches
+ // only compares primitive types. A DECIMAL bound decoded at the
file's scale would then be
+ // compared against the table's scale as if the payloads shared a
domain, so leave the zone
+ // map out unless the two types agree exactly.
+ if (!expr_zonemap::data_types_compatible(col_schema->data_type,
slot->type())) {
+ ctx.slots.emplace(cid, std::move(slot_zone_map));
+ continue;
+ }
int parquet_col_id = col_schema->physical_column_index;
if (parquet_col_id < 0) {
// Complex parent fields do not map to a physical Parquet column.
@@ -1578,7 +1609,7 @@ Status ParquetReader::_process_expr_zonemap_filter(const
tparquet::RowGroup& row
ctx.slots.emplace(cid, std::move(slot_zone_map));
}
- const auto result = VExprContext::evaluate_zonemap_filter(all_conjuncts,
ctx);
+ const auto result =
VExprContext::evaluate_zonemap_filter(single_slot_conjuncts, ctx);
ctx.stats.accumulate_to(&_reader_statistics);
if (result == ZoneMapFilterResult::kNoMatch) {
*filter_group = true;
diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp
b/be/src/format_v2/parquet/parquet_statistics.cpp
index e4928e65a7c..6895c9743c7 100644
--- a/be/src/format_v2/parquet/parquet_statistics.cpp
+++ b/be/src/format_v2/parquet/parquet_statistics.cpp
@@ -904,11 +904,8 @@ void add_slot_zonemap(ZoneMapEvalContext* ctx, int
slot_index, const DataTypePtr
std::shared_ptr<segment_v2::ZoneMap> zone_map) {
DORIS_CHECK(ctx != nullptr);
ZoneMapEvalContext::SlotZoneMap slot_zone_map;
- slot_zone_map.data_type = data_type;
+ slot_zone_map.set_data_type_from_parquet(data_type);
slot_zone_map.zone_map = std::move(zone_map);
- const auto primitive_type =
remove_nullable(data_type)->get_primitive_type();
- slot_zone_map.floating_nan_count_unknown =
- primitive_type == TYPE_FLOAT || primitive_type == TYPE_DOUBLE;
ctx->slots.emplace(slot_index, std::move(slot_zone_map));
}
diff --git a/be/src/storage/index/zone_map/zonemap_eval_context.h
b/be/src/storage/index/zone_map/zonemap_eval_context.h
index 7c1afab8780..25eb935ac44 100644
--- a/be/src/storage/index/zone_map/zonemap_eval_context.h
+++ b/be/src/storage/index/zone_map/zonemap_eval_context.h
@@ -22,6 +22,7 @@
#include <memory>
#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_nullable.h"
#include "storage/index/zone_map/zone_map_index.h"
#include "storage/index/zone_map/zonemap_filter_result.h"
@@ -54,6 +55,22 @@ public:
std::shared_ptr<const segment_v2::ZoneMap> zone_map;
// Parquet min/max does not expose whether a floating chunk also
contains NaNs.
bool floating_nan_count_unknown = false;
+
+ // Every caller that fills this from Parquet statistics must go
through here. Setting
+ // data_type without the flag silently re-enables pruning that Parquet
bounds cannot
+ // support, so the two fields are assigned together on purpose.
+ void set_data_type_from_parquet(const DataTypePtr& type) {
+ data_type = type;
+ // SlotDescriptor's test-only default constructor leaves the type
null, and the callers
+ // used to just copy it through.
+ floating_nan_count_unknown = false;
+ if (type == nullptr) {
+ return;
+ }
+ const auto primitive_type =
remove_nullable(type)->get_primitive_type();
+ floating_nan_count_unknown =
+ primitive_type == TYPE_FLOAT || primitive_type ==
TYPE_DOUBLE;
+ }
};
std::shared_ptr<const segment_v2::ZoneMap> zone_map(int slot_index) const;
diff --git a/be/src/storage/segment/segment.cpp
b/be/src/storage/segment/segment.cpp
index ca7db7148b7..90bddf887f6 100644
--- a/be/src/storage/segment/segment.cpp
+++ b/be/src/storage/segment/segment.cpp
@@ -118,6 +118,12 @@ Status build_segment_zonemap_context(Segment* segment,
const Schema& schema,
continue;
}
const auto column_id = schema.column_id(cast_set<size_t>(slot_index));
+ // Single-version rowsets substitute the version after reading in 4.2.
Its on-disk zone
+ // map holds only a placeholder, so it cannot prove that the
expression will not match.
+ if (read_options.version.first == read_options.version.second &&
+ cast_set<int32_t>(column_id) == schema.version_col_idx()) {
+ continue;
+ }
const auto* tablet_column = schema.column(column_id);
DORIS_CHECK(tablet_column != nullptr);
if (!segment->can_apply_predicate_safely(
diff --git a/be/src/storage/segment/segment_iterator.cpp
b/be/src/storage/segment/segment_iterator.cpp
index 2ec15cb8533..6213aa265c4 100644
--- a/be/src/storage/segment/segment_iterator.cpp
+++ b/be/src/storage/segment/segment_iterator.cpp
@@ -3723,6 +3723,12 @@ Status
SegmentIterator::_apply_expr_zonemap_to_row_ranges(const VExprContextSPtr
continue;
}
const auto cid = _schema->column_id(cast_set<size_t>(slot_index));
+ // Match the segment-level exclusion: the version is substituted after
the storage read,
+ // while page zone maps still describe the placeholder stored on disk.
+ if (_opts.version.first == _opts.version.second &&
+ cast_set<int32_t>(cid) == _schema->version_col_idx()) {
+ continue;
+ }
if (!_segment->can_apply_predicate_safely(cid, *_schema,
_opts.target_cast_type_for_variants, _opts)) {
continue;
diff --git a/be/test/exprs/expr_zonemap_filter_test.cpp
b/be/test/exprs/expr_zonemap_filter_test.cpp
index 03b5159262b..1e3909d1308 100644
--- a/be/test/exprs/expr_zonemap_filter_test.cpp
+++ b/be/test/exprs/expr_zonemap_filter_test.cpp
@@ -33,13 +33,17 @@
#include "common/object_pool.h"
#include "core/column/column_vector.h"
#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_bitmap.h"
#include "core/data_type/data_type_date_or_datetime_v2.h"
#include "core/data_type/data_type_decimal.h"
+#include "core/data_type/data_type_factory.hpp"
+#include "core/data_type/data_type_jsonb.h"
#include "core/data_type/data_type_nullable.h"
#include "core/data_type/data_type_number.h"
#include "core/data_type/data_type_string.h"
#include "core/data_type/data_type_struct.h"
#include "core/data_type/data_type_varbinary.h"
+#include "core/data_type/data_type_variant.h"
#include "core/field.h"
#include "core/string_ref.h"
#include "core/value/vdatetime_value.h"
@@ -142,6 +146,23 @@ ZoneMapEvalContext make_context(segment_v2::ZoneMap
zone_map, const DataTypePtr&
return ctx;
}
+// Two slots in one context, which is what the segment-level and Parquet
row-group-level callers
+// build for a compound expression. Slot 0 is the left operand, slot 1 the
right.
+ZoneMapEvalContext make_two_slot_context(segment_v2::ZoneMap left,
segment_v2::ZoneMap right,
+ const DataTypePtr& left_type,
+ const DataTypePtr& right_type) {
+ ZoneMapEvalContext ctx;
+ ZoneMapEvalContext::SlotZoneMap left_slot;
+ left_slot.data_type = left_type;
+ left_slot.zone_map =
std::make_shared<segment_v2::ZoneMap>(std::move(left));
+ ctx.slots.emplace(0, std::move(left_slot));
+ ZoneMapEvalContext::SlotZoneMap right_slot;
+ right_slot.data_type = right_type;
+ right_slot.zone_map =
std::make_shared<segment_v2::ZoneMap>(std::move(right));
+ ctx.slots.emplace(1, std::move(right_slot));
+ return ctx;
+}
+
DictionaryEvalContext make_dictionary_context(std::vector<Field> values,
const DataTypePtr& data_type) {
DictionaryEvalContext ctx;
@@ -308,6 +329,28 @@ private:
std::string _expr_name = "unsupported_single_slot_expr";
};
+// A stand-in for a binary comparison predicate node (`a < b`, `a = 5`, ...).
+// contains_slot_slot_comparison inspects only children(), so the concrete
node type does not matter:
+// a comparison's two operands are its children.
+class BinaryLeafExpr final : public VExpr {
+public:
+ BinaryLeafExpr(VExprSPtr left, VExprSPtr right)
+ : VExpr(std::make_shared<DataTypeUInt8>(), false) {
+ add_child(std::move(left));
+ add_child(std::move(right));
+ }
+
+ const std::string& expr_name() const override { return _expr_name; }
+
+ Status execute_column_impl(VExprContext*, const Block*, const Selector*,
size_t,
+ ColumnPtr&) const override {
+ return Status::InternalError("BinaryLeafExpr is only used by zonemap
tests");
+ }
+
+private:
+ std::string _expr_name = "binary_leaf_expr";
+};
+
VExprSPtr make_fixed_zonemap_expr(ZoneMapFilterResult result) {
return std::make_shared<FixedZonemapExpr>(result);
}
@@ -1478,4 +1521,426 @@ TEST(ExprZonemapFilterTest,
ExprContextZonemapEvaluationShortCircuitsOnNoMatch)
EXPECT_EQ(0, ctx.stats.unusable_zonemap_eval_count);
}
+TEST(ExprZonemapFilterTest,
SlotSlotNotEqualsPrunesOnlyWhenBothCollapseToOneValue) {
+ auto type = int_type();
+ auto left = make_slot(0, type);
+ auto right = make_slot(1, type);
+ FunctionComparison<NotEqualsOp, NameNotEquals> not_equals;
+
+ // Both columns hold the single value 5, so `a != b` is false on every row.
+ auto same_point =
+ make_two_slot_context(make_int_zonemap(5, 5), make_int_zonemap(5,
5), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ not_equals.evaluate_zonemap_filter(same_point, {left, right}));
+
+ // Two different single values: `a != b` is true everywhere, so nothing
can be pruned.
+ auto other_point =
+ make_two_slot_context(make_int_zonemap(5, 5), make_int_zonemap(6,
6), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ not_equals.evaluate_zonemap_filter(other_point, {left, right}));
+
+ // A non-degenerate range on either side leaves the outcome unknown.
+ auto left_range =
+ make_two_slot_context(make_int_zonemap(5, 6), make_int_zonemap(5,
5), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ not_equals.evaluate_zonemap_filter(left_range, {left, right}));
+ auto right_range =
+ make_two_slot_context(make_int_zonemap(5, 5), make_int_zonemap(5,
6), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ not_equals.evaluate_zonemap_filter(right_range, {left, right}));
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotCoversAllSixOperatorsAtTheBoundary) {
+ auto type = int_type();
+ auto left = make_slot(0, type);
+ auto right = make_slot(1, type);
+ // Left entirely above right, ranges touching at 10: [10, 20] vs [1, 10].
+ auto ctx = make_two_slot_context(make_int_zonemap(10, 20),
make_int_zonemap(1, 10), type, type);
+
+ // a < b needs min_a >= max_b, which holds at the touching bound.
+ FunctionComparison<LessOp, NameLess> less;
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch, less.evaluate_zonemap_filter(ctx,
{left, right}));
+ // a <= b needs min_a > max_b, which does not hold: a == b == 10 is
possible.
+ FunctionComparison<LessOrEqualsOp, NameLessOrEquals> less_equals;
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ less_equals.evaluate_zonemap_filter(ctx, {left, right}));
+ FunctionComparison<GreaterOp, NameGreater> greater;
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
greater.evaluate_zonemap_filter(ctx, {left, right}));
+ FunctionComparison<EqualsOp, NameEquals> equals;
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
equals.evaluate_zonemap_filter(ctx, {left, right}));
+
+ // Fully disjoint: equality becomes impossible and so does a <= b.
+ auto disjoint =
+ make_two_slot_context(make_int_zonemap(10, 20),
make_int_zonemap(1, 9), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ equals.evaluate_zonemap_filter(disjoint, {left, right}));
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ less_equals.evaluate_zonemap_filter(disjoint, {left, right}));
+
+ // Mirrored, so a > b and a >= b become impossible.
+ auto mirrored =
+ make_two_slot_context(make_int_zonemap(1, 9), make_int_zonemap(10,
20), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ greater.evaluate_zonemap_filter(mirrored, {left, right}));
+ FunctionComparison<GreaterOrEqualsOp, NameGreaterOrEquals> greater_equals;
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ greater_equals.evaluate_zonemap_filter(mirrored, {left, right}));
+
+ // Partially overlapping ranges: every operator must keep the zone. These
are the cases that
+ // pin down which end of each range the rule reads. Taking max_a instead
of min_a for a < b, or
+ // max_b instead of min_b for a > b, still prunes here and would drop the
overlapping rows.
+ // [10, 20] vs [5, 15] overlap on [10, 15]: a = 12, b = 14 satisfies a < b
and a <= b.
+ auto left_high =
+ make_two_slot_context(make_int_zonemap(10, 20),
make_int_zonemap(5, 15), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ less.evaluate_zonemap_filter(left_high, {left, right}));
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ less_equals.evaluate_zonemap_filter(left_high, {left, right}));
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ equals.evaluate_zonemap_filter(left_high, {left, right}));
+ // [5, 15] vs [10, 20]: a = 14, b = 12 satisfies a > b and a >= b.
+ auto right_high =
+ make_two_slot_context(make_int_zonemap(5, 15),
make_int_zonemap(10, 20), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ greater.evaluate_zonemap_filter(right_high, {left, right}));
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ greater_equals.evaluate_zonemap_filter(right_high, {left,
right}));
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ equals.evaluate_zonemap_filter(right_high, {left, right}));
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotGuardsCoverNullAndMissingStatistics) {
+ auto type = int_type();
+ auto left = make_slot(0, type);
+ auto right = make_slot(1, type);
+ FunctionComparison<LessOp, NameLess> less;
+
+ // A column with no non-null value makes the comparison NULL on every row.
Production builds
+ // such a zone map with default-constructed min/max, which carry TYPE_NULL
rather than the
+ // column type. That is why the has_not_null check has to run before the
range check: the range
+ // check asserts the bounds match the slot type and would fatal on
TYPE_NULL.
+ segment_v2::ZoneMap all_null_zonemap;
+ all_null_zonemap.has_null = true;
+ all_null_zonemap.has_not_null = false;
+ auto left_all_null =
+ make_two_slot_context(all_null_zonemap, make_int_zonemap(1, 9),
type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ less.evaluate_zonemap_filter(left_all_null, {left, right}));
+ EXPECT_EQ(0, left_all_null.stats.unusable_zonemap_eval_count);
+ // The same on the right operand: both halves of the check are
load-bearing.
+ auto right_all_null =
+ make_two_slot_context(make_int_zonemap(1, 9), all_null_zonemap,
type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ less.evaluate_zonemap_filter(right_all_null, {left, right}));
+ EXPECT_EQ(0, right_all_null.stats.unusable_zonemap_eval_count);
+
+ // The right slot is absent from the context entirely.
+ auto missing_slot = make_context(make_int_zonemap(1, 9), type);
+ EXPECT_EQ(ZoneMapFilterResult::kUnsupported,
+ less.evaluate_zonemap_filter(missing_slot, {left, right}));
+ EXPECT_EQ(1, missing_slot.stats.unusable_zonemap_eval_count);
+
+ // The right slot is present with a type but without a zone map.
+ auto missing_zonemap = make_context(make_int_zonemap(1, 9), type);
+ ZoneMapEvalContext::SlotZoneMap slot_without_zonemap;
+ slot_without_zonemap.data_type = type;
+ missing_zonemap.slots.emplace(1, std::move(slot_without_zonemap));
+ EXPECT_EQ(ZoneMapFilterResult::kUnsupported,
+ less.evaluate_zonemap_filter(missing_zonemap, {left, right}));
+ EXPECT_EQ(1, missing_zonemap.stats.unusable_zonemap_eval_count);
+
+ // pass_all marks the bounds as unusable even though has_not_null is set.
+ auto pass_all_zonemap = make_int_zonemap(1, 9);
+ pass_all_zonemap.pass_all = true;
+ auto pass_all = make_two_slot_context(make_int_zonemap(10, 20),
std::move(pass_all_zonemap),
+ type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kUnsupported,
+ less.evaluate_zonemap_filter(pass_all, {left, right}));
+ EXPECT_EQ(1, pass_all.stats.unusable_zonemap_eval_count);
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotHandlesTheSameSlotOnBothSides) {
+ // FE's SimplifySelfComparison rewrites `a != a` (parsed as Not(EqualTo(a,
a))) by folding the
+ // inner equality, so it does not normally reach the BE as a slot-vs-slot
NotEqualTo. This pins
+ // the conservative fallback if such a shape still reaches the evaluator:
the generic point-range
+ // rule proves kNoMatch when a is a single value, and a range stays
kMayMatch instead of being
+ // special-cased as self-comparison.
+ auto type = int_type();
+ auto slot = make_slot(0, type);
+ FunctionComparison<NotEqualsOp, NameNotEquals> not_equals;
+
+ auto single_value = make_context(make_int_zonemap(7, 7), type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ not_equals.evaluate_zonemap_filter(single_value, {slot, slot}));
+ auto range = make_context(make_int_zonemap(7, 8), type);
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ not_equals.evaluate_zonemap_filter(range, {slot, slot}));
+}
+
+TEST(ExprZonemapFilterTest, ContainsSlotSlotComparisonFindsSlotPairsAnywhere) {
+ auto type = int_type();
+ auto slot_a = make_slot(0, type);
+ auto slot_b = make_slot(1, type);
+
+ EXPECT_FALSE(expr_zonemap::contains_slot_slot_comparison(nullptr));
+
+ // `a < b`: two distinct slots at the root.
+ EXPECT_TRUE(expr_zonemap::contains_slot_slot_comparison(
+ std::make_shared<BinaryLeafExpr>(slot_a, slot_b)));
+
+ // `a < a`: same column on both sides. This is the case a distinct-column
count misses -- one
+ // column id, still a slot-vs-slot leaf -- and the regression this gate
has to catch.
+ EXPECT_TRUE(expr_zonemap::contains_slot_slot_comparison(
+ std::make_shared<BinaryLeafExpr>(slot_a, slot_a)));
+
+ // `a < 5`: slot versus literal is not a slot-vs-slot comparison.
+ EXPECT_FALSE(expr_zonemap::contains_slot_slot_comparison(
+ std::make_shared<BinaryLeafExpr>(slot_a, make_int_literal(5))));
+
+ // `a < 5 AND c < d`: the slot pair is nested on the second child of the
AND.
+ auto slot_c = make_slot(2, type);
+ auto slot_d = make_slot(3, type);
+ auto and_pred =
+
std::make_shared<VCompoundPred>(make_compound_node(TExprOpcode::COMPOUND_AND,
2));
+ and_pred->add_child(std::make_shared<BinaryLeafExpr>(slot_a,
make_int_literal(5)));
+ and_pred->add_child(std::make_shared<BinaryLeafExpr>(slot_c, slot_d));
+ EXPECT_TRUE(expr_zonemap::contains_slot_slot_comparison(and_pred));
+
+ // `a < 5 OR b < 6`: only slot-vs-literal leaves, so no slot pair anywhere
in the tree.
+ auto or_pred =
std::make_shared<VCompoundPred>(make_compound_node(TExprOpcode::COMPOUND_OR,
2));
+ or_pred->add_child(std::make_shared<BinaryLeafExpr>(slot_a,
make_int_literal(5)));
+ or_pred->add_child(std::make_shared<BinaryLeafExpr>(slot_b,
make_int_literal(6)));
+ EXPECT_FALSE(expr_zonemap::contains_slot_slot_comparison(or_pred));
+}
+
+TEST(ExprZonemapFilterTest,
SlotSlotWidensOnlyTheZonemapGateNotDictionaryOrBloom) {
+ // This is the regression guard for the trap in this change:
comparison_zonemap_detail's
+ // can_evaluate is shared by dictionary filtering and
can_evaluate_equality, and both dereference
+ // extract_slot_and_literal behind a DORIS_CHECK. Widening it would abort
on `a != b`, so the
+ // slot-vs-slot shape must only ever be accepted by
can_evaluate_zonemap_filter.
+ auto type = int_type();
+ auto left = make_slot(0, type);
+ auto right = make_slot(1, type);
+ FunctionComparison<NotEqualsOp, NameNotEquals> not_equals;
+
+ EXPECT_TRUE(not_equals.can_evaluate_zonemap_filter({left, right}));
+ EXPECT_FALSE(not_equals.can_evaluate_dictionary_filter({left, right}));
+
+ // The bloom gate has to be checked through EQ. can_evaluate_bloom_filter
starts with
+ // `op == Op::EQ`, so asserting it on NE would pass no matter what the
rest of the gate does.
+ FunctionComparison<EqualsOp, NameEquals> equals;
+ EXPECT_TRUE(equals.can_evaluate_zonemap_filter({left, right}));
+ EXPECT_FALSE(equals.can_evaluate_bloom_filter({left, right}));
+ EXPECT_TRUE(equals.can_evaluate_bloom_filter({left, make_int_literal(1)}));
+
+ // Slot vs literal keeps working on every gate.
+ EXPECT_TRUE(not_equals.can_evaluate_zonemap_filter({left,
make_int_literal(1)}));
+ EXPECT_TRUE(not_equals.can_evaluate_dictionary_filter({left,
make_int_literal(1)}));
+
+ // Two columns of incompatible types are rejected: the two Fields would be
compared directly and
+ // Field comparison throws on mismatched non-string types.
+ auto bigint = std::make_shared<DataTypeInt64>();
+ EXPECT_FALSE(not_equals.can_evaluate_zonemap_filter({left, make_slot(1,
bigint)}));
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotGateRejectsVarbinaryButAllowsStrings) {
+ // VARBINARY is rejected at the gate: the v1 Parquet path produces a
TYPE_STRING bound for a
+ // VARBINARY slot, so evaluating it would abort in
fetch_compatible_slot_type instead of falling
+ // back. String and int pairs are admitted; the string bound-length safety
is in evaluation.
+ auto string_type = std::make_shared<DataTypeString>();
+ auto varbinary_type = std::make_shared<DataTypeVarbinary>();
+ FunctionComparison<GreaterOp, NameGreater> greater;
+
+ EXPECT_FALSE(greater.can_evaluate_zonemap_filter(
+ {make_slot(0, varbinary_type), make_slot(1, varbinary_type)}));
+
+ EXPECT_TRUE(greater.can_evaluate_zonemap_filter(
+ {make_slot(0, string_type), make_slot(1, string_type)}));
+ auto type = int_type();
+ EXPECT_TRUE(greater.can_evaluate_zonemap_filter({make_slot(0, type),
make_slot(1, type)}));
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotGateRejectsVariantAndUnorderedTypes) {
+ // A segment types a VARIANT path slot by its inferred leaf type, so
admitting a VARIANT pair
+ // would fail fetch_compatible_slot_type's check. Complex, JSONB and
aggregate-state Fields all
+ // compare equal, so their bounds prove nothing. Each is rejected,
nullable or not.
+ FunctionComparison<EqualsOp, NameEquals> equals;
+ const std::vector<DataTypePtr> excluded_types = {
+ std::make_shared<DataTypeVariant>(),
+ std::make_shared<DataTypeJsonb>(),
+ std::make_shared<DataTypeArray>(make_nullable(int_type())),
+ std::make_shared<DataTypeBitMap>(),
+ };
+ for (const auto& excluded : excluded_types) {
+ for (const auto& type : {excluded, make_nullable(excluded)}) {
+ EXPECT_FALSE(
+ equals.can_evaluate_zonemap_filter({make_slot(0, type),
make_slot(1, type)}))
+ << type->get_name();
+ }
+ }
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotStringPrunesUntruncatedBoundsOnly) {
+ // A native string zone-map max of exactly 512 bytes was bumped upward and
may also have been
+ // truncated, so it may not be a real upper bound. A shorter bound is the
exact value.
+ // left in ["a","c"], right pinned above "c", so `left > right` cannot
hold when the bounds
+ // are trusted.
+ auto string_type = std::make_shared<DataTypeString>();
+ auto left = make_slot(0, string_type);
+ auto right = make_slot(1, string_type);
+ FunctionComparison<GreaterOp, NameGreater> greater;
+
+ auto eval = [&](const std::string& r) {
+ auto ctx = make_two_slot_context(make_string_zonemap("a", "c"),
make_string_zonemap(r, r),
+ string_type, string_type);
+ return greater.evaluate_zonemap_filter(ctx, {left, right});
+ };
+
+ // Short and 511-byte bounds are untruncated, so pruning stands.
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch, eval("m"));
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch, eval(std::string(511, 'x')));
+ // A 512-byte bound may be truncated/bumped, and a longer one is out of
spec: fall back.
+ EXPECT_EQ(ZoneMapFilterResult::kUnsupported, eval(std::string(512, 'x')));
+ EXPECT_EQ(ZoneMapFilterResult::kUnsupported, eval(std::string(513, 'x')));
+ // Before the carry fix the writer bumped only the last byte, so a 0xff
there wrapped to 0x00
+ // and left a 512-byte bound ending in '\0' below the data; still length
512, still unusable.
+ std::string wrapped(511, 'x');
+ wrapped.push_back('\0');
+ EXPECT_EQ(ZoneMapFilterResult::kUnsupported, eval(wrapped));
+
+ // An embedded NUL counts toward .size(); a short bound with one stays
usable.
+ std::string with_nul("a\0b", 3);
+ auto ctx_nul = make_two_slot_context(make_string_zonemap(with_nul,
with_nul),
+ make_string_zonemap("m", "z"),
string_type, string_type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ greater.evaluate_zonemap_filter(ctx_nul, {left, right}));
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotBailsOutWhenAFloatingNanCountIsUnknown) {
+ // Parquet bounds omit NaN without reporting how many were skipped, so a
zone map that looks like
+ // a single point may still hide one. That would flip the NE rule from
false to true, so the
+ // slot-vs-slot path refuses to prune for every operator once either side
is marked unknown.
+ auto type = std::make_shared<DataTypeFloat64>();
+ auto left = make_slot(0, type);
+ auto right = make_slot(1, type);
+
+ auto make_double_zonemap = [](double value) {
+ segment_v2::ZoneMap zone_map;
+ zone_map.min_value = Field::create_field<TYPE_DOUBLE>(value);
+ zone_map.max_value = Field::create_field<TYPE_DOUBLE>(value);
+ zone_map.has_not_null = true;
+ return zone_map;
+ };
+
+ // Control group. Both sides are the same single value, so NE prunes
without the guard; the
+ // contrast with the loop below is what shows the guard is what changes
the outcome. The counter
+ // alone cannot show that, since every bail-out path in evaluate_slot_slot
bumps the same one.
+ FunctionComparison<NotEqualsOp, NameNotEquals> not_equals;
+ auto known =
+ make_two_slot_context(make_double_zonemap(1.0),
make_double_zonemap(1.0), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ not_equals.evaluate_zonemap_filter(known, {left, right}));
+ EXPECT_EQ(0, known.stats.unusable_zonemap_eval_count);
+
+ auto expect_bails_out = [&](auto&& comparison, const char* predicate) {
+ for (int unknown_slot : {0, 1}) {
+ auto ctx = make_two_slot_context(make_double_zonemap(1.0),
make_double_zonemap(1.0),
+ type, type);
+ ctx.slots[unknown_slot].floating_nan_count_unknown = true;
+ EXPECT_EQ(ZoneMapFilterResult::kUnsupported,
+ comparison.evaluate_zonemap_filter(ctx, {left, right}))
+ << predicate << " must stop pruning when slot " <<
unknown_slot
+ << " has an unknown NaN count";
+ EXPECT_EQ(1, ctx.stats.unusable_zonemap_eval_count);
+ }
+ };
+
+ // All six operators bail out. On these bounds NE, LT and GT would
otherwise prune and EQ, LE
+ // and GE would otherwise keep the zone, so none of the six is vacuously
covered.
+ expect_bails_out(FunctionComparison<EqualsOp, NameEquals> {}, "a = b");
+ expect_bails_out(FunctionComparison<NotEqualsOp, NameNotEquals> {}, "a !=
b");
+ expect_bails_out(FunctionComparison<LessOp, NameLess> {}, "a < b");
+ expect_bails_out(FunctionComparison<LessOrEqualsOp, NameLessOrEquals> {},
"a <= b");
+ expect_bails_out(FunctionComparison<GreaterOp, NameGreater> {}, "a > b");
+ expect_bails_out(FunctionComparison<GreaterOrEqualsOp,
NameGreaterOrEquals> {}, "a >= b");
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotComparesDecimalBoundsInOneDomain) {
+ // Decimal bounds are unscaled payloads, so the rule is only meaningful
when both sides share a
+ // scale. The v1 Parquet reader decodes row-group bounds in the file's own
logical type while
+ // _type_matches compares only primitive types, so it now leaves the zone
map out when the file
+ // and table types differ; this pins the behaviour the rule needs when
they agree.
+ auto decimal_type =
DataTypeFactory::instance().create_data_type(TYPE_DECIMAL64, false, 18, 2);
+ auto left = make_slot(0, decimal_type);
+ auto right = make_slot(1, decimal_type);
+ FunctionComparison<LessOp, NameLess> less;
+
+ auto make_decimal_zonemap = [](int64_t unscaled_min, int64_t unscaled_max)
{
+ segment_v2::ZoneMap zone_map;
+ zone_map.min_value =
Field::create_field<TYPE_DECIMAL64>(Decimal64(unscaled_min));
+ zone_map.max_value =
Field::create_field<TYPE_DECIMAL64>(Decimal64(unscaled_max));
+ zone_map.has_not_null = true;
+ return zone_map;
+ };
+
+ // 10.00 .. 20.00 against 0.50 .. 1.00: a < b cannot hold.
+ auto separated =
+ make_two_slot_context(make_decimal_zonemap(1000, 2000),
make_decimal_zonemap(50, 100),
+ decimal_type, decimal_type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ less.evaluate_zonemap_filter(separated, {left, right}));
+
+ // Touching at 10.00, which is what separates `lmin >= rmax` from `lmin >
rmax`: a < b still
+ // cannot hold, so LT must prune while LE must not.
+ auto touching =
+ make_two_slot_context(make_decimal_zonemap(1000, 2000),
make_decimal_zonemap(500, 1000),
+ decimal_type, decimal_type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
less.evaluate_zonemap_filter(touching, {left, right}));
+ FunctionComparison<LessOrEqualsOp, NameLessOrEquals> less_equals;
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ less_equals.evaluate_zonemap_filter(touching, {left, right}));
+
+ // Overlapping in the same scale: nothing can be concluded.
+ auto overlapping =
+ make_two_slot_context(make_decimal_zonemap(1000, 2000),
+ make_decimal_zonemap(1500, 2500),
decimal_type, decimal_type);
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ less.evaluate_zonemap_filter(overlapping, {left, right}));
+}
+
+TEST(ExprZonemapFilterTest, ParquetSlotZoneMapMarksFloatingNanCountUnknown) {
+ // The rule used to be spelled out in format_v2's add_slot_zonemap only,
and the v1 Parquet
+ // reader assigned data_type directly and so left the flag false. Two
double columns whose
+ // Parquet bounds both read [1.0, 1.0] then got pruned by `a != b` even
when one of them hid a
+ // NaN. Both readers now go through this setter, so this test pins the
rule down in one place.
+ ZoneMapEvalContext::SlotZoneMap double_slot;
+
double_slot.set_data_type_from_parquet(std::make_shared<DataTypeFloat64>());
+ EXPECT_TRUE(double_slot.floating_nan_count_unknown);
+
+ ZoneMapEvalContext::SlotZoneMap float_slot;
+ float_slot.set_data_type_from_parquet(std::make_shared<DataTypeFloat32>());
+ EXPECT_TRUE(float_slot.floating_nan_count_unknown);
+
+ // Nullable wrappers must not hide the floating type.
+ ZoneMapEvalContext::SlotZoneMap nullable_double_slot;
+ nullable_double_slot.set_data_type_from_parquet(
+
std::make_shared<DataTypeNullable>(std::make_shared<DataTypeFloat64>()));
+ EXPECT_TRUE(nullable_double_slot.floating_nan_count_unknown);
+
+ // Non-floating columns keep exact bounds, so they must stay prunable.
+ ZoneMapEvalContext::SlotZoneMap int_slot;
+ auto int_data_type = int_type();
+ int_slot.set_data_type_from_parquet(int_data_type);
+ EXPECT_FALSE(int_slot.floating_nan_count_unknown);
+ EXPECT_EQ(int_data_type, int_slot.data_type);
+
+ // A slot with no type at all is copied through as before rather than
dereferenced. The type is
+ // null only for SlotDescriptor's test-only default constructor, and the
evaluator treats a slot
+ // without a type as unusable anyway.
+ ZoneMapEvalContext::SlotZoneMap untyped_slot;
+ untyped_slot.set_data_type_from_parquet(nullptr);
+ EXPECT_EQ(nullptr, untyped_slot.data_type);
+ EXPECT_FALSE(untyped_slot.floating_nan_count_unknown);
+}
+
} // namespace doris
diff --git
a/regression-test/data/query_p0/expr_zonemap/test_expr_zonemap_pruning.out
b/regression-test/data/query_p0/expr_zonemap/test_expr_zonemap_pruning.out
new file mode 100644
index 00000000000..9a0e3e3a4fb
--- /dev/null
+++ b/regression-test/data/query_p0/expr_zonemap/test_expr_zonemap_pruning.out
@@ -0,0 +1,124 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !matched_rows --
+4096
+
+-- !starts_with_pruned --
+0
+
+-- !in_pruned --
+0
+
+-- !is_null_pruned --
+0
+
+-- !comparison_pruned --
+0
+
+-- !page_reachability --
+4096
+
+-- !inferred_prefix_range --
+4096
+
+-- !inferred_date_range --
+4096
+
+-- !inferred_date_format --
+4096
+
+-- !inferred_date_format_equality --
+4096
+
+-- !inferred_direct_scan --
+4096
+
+-- !large_quarter_period --
+8192
+
+-- !char_pruned --
+0
+
+-- !is_not_null_pruned --
+0
+
+-- !two_column_gt --
+0
+
+-- !two_column_ge --
+0
+
+-- !two_column_lt --
+0
+
+-- !two_column_le --
+0
+
+-- !two_column_eq --
+0
+
+-- !two_column_ne --
+0
+
+-- !overlap_lo_lt_alt --
+2048
+
+-- !overlap_lo_ne_alt --
+2048
+
+-- !overlap_lo_eq_alt --
+2048
+
+-- !overlap_lo_le_alt --
+4096
+
+-- !overlap_lo_gt_alt --
+0
+
+-- !overlap_lo_ge_alt_shift --
+4096
+
+-- !overlap_lo_lt_cast_hi --
+4096
+
+-- !two_column_null --
+0
+
+-- !two_column_all_null --
+0
+
+-- !per_segment_ne --
+1024
+
+-- !per_segment_eq --
+2048
+
+-- !per_segment_ne_without --
+1024
+
+-- !per_segment_eq_without --
+2048
+
+-- !literal_still_prunes --
+0
+
+-- !column_predicate --
+0
+
+-- !double_pruned --
+0
+
+-- !nan_lo_gt_hi --
+3
+
+-- !nan_lo_ne_hi --
+6
+
+-- !nan_lo_eq_hi --
+0
+
+-- !nan_lo_lt_hi --
+3
+
+-- !hidden_version_or_miss --
+2048
+
diff --git
a/regression-test/suites/query_p0/expr_zonemap/test_expr_zonemap_pruning.groovy
b/regression-test/suites/query_p0/expr_zonemap/test_expr_zonemap_pruning.groovy
index 7c468a0d1bf..1e280bfb45e 100644
---
a/regression-test/suites/query_p0/expr_zonemap/test_expr_zonemap_pruning.groovy
+++
b/regression-test/suites/query_p0/expr_zonemap/test_expr_zonemap_pruning.groovy
@@ -49,16 +49,11 @@ suite("test_expr_zonemap_pruning") {
def profileAction = new ProfileAction(context)
def getProfileByToken = { String token ->
- for (int retry = 0; retry < 10; ++retry) {
- List profileData = profileAction.getProfileList()
- for (final def profileItem in profileData) {
- if (profileItem["Sql Statement"].toString().contains(token)) {
- return profileAction.getProfile(profileItem["Profile
ID"].toString())
- }
- }
- Thread.sleep(500)
- }
- throw new IllegalStateException("Missing profile for token: " + token)
+ // Wait for the profile to reach COMPLETE before reading counters. A
profile can be listed and
+ // fetchable while still aggregating across instances, so an exact
counter (the 2 / 1 / 3
+ // filtered-segment assertions below) could otherwise read a partial
total. getProfileBySql
+ // polls Profile Completion State and returns only once it is COMPLETE.
+ return profileAction.getProfileBySql(token)
}
def counterSum = { String profile, String counterName ->
@@ -105,40 +100,50 @@ suite("test_expr_zonemap_pruning") {
assertTrue(counterValue > 0)
}
- def matchedRows = sql """
+ qt_matched_rows """
SELECT COUNT(*) FROM test_expr_zonemap_pruning WHERE starts_with(v,
'a')
"""
- assertEquals(4096L, matchedRows[0][0] as long)
+ qt_starts_with_pruned """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning WHERE starts_with(v,
'z')
+ """
def startsWithToken = "expr_zonemap_pruning_starts_with_" +
UUID.randomUUID().toString()
- def startsWithPrunedRows = sql """
+ sql """
SELECT '${startsWithToken}', COUNT(*) FROM test_expr_zonemap_pruning
WHERE starts_with(v, 'z')
"""
- assertEquals(0L, startsWithPrunedRows[0][1] as long)
assertExprZonemapPruned(startsWithToken)
+ qt_in_pruned """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning
+ WHERE v IN ('z0', 'z1') OR starts_with(v, 'zz')
+ """
def inToken = "expr_zonemap_pruning_in_" + UUID.randomUUID().toString()
- def inPrunedRows = sql """
+ sql """
SELECT '${inToken}', COUNT(*) FROM test_expr_zonemap_pruning
WHERE v IN ('z0', 'z1') OR starts_with(v, 'zz')
"""
- assertEquals(0L, inPrunedRows[0][1] as long)
assertExprZonemapPruned(inToken)
+ qt_is_null_pruned """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning
+ WHERE v IS NULL OR starts_with(v, 'zz')
+ """
def nullToken = "expr_zonemap_pruning_is_null_" +
UUID.randomUUID().toString()
- def nullPrunedRows = sql """
+ sql """
SELECT '${nullToken}', COUNT(*) FROM test_expr_zonemap_pruning
WHERE v IS NULL OR starts_with(v, 'zz')
"""
- assertEquals(0L, nullPrunedRows[0][1] as long)
assertExprZonemapPruned(nullToken)
+ qt_comparison_pruned """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning
+ WHERE id > 5000 OR id < 0
+ """
def comparisonToken = "expr_zonemap_pruning_comparison_" +
UUID.randomUUID().toString()
- def comparisonPrunedRows = sql """
+ sql """
SELECT '${comparisonToken}', COUNT(*) FROM test_expr_zonemap_pruning
WHERE id > 5000 OR id < 0
"""
- assertEquals(0L, comparisonPrunedRows[0][1] as long)
assertExprZonemapPruned(comparisonToken)
sql """ DROP TABLE IF EXISTS test_expr_zonemap_page_reachability """
@@ -166,12 +171,15 @@ suite("test_expr_zonemap_pruning") {
"""
sql """ sync """
+ qt_page_reachability """
+ SELECT COUNT(*) FROM test_expr_zonemap_page_reachability
+ WHERE starts_with(v, 'm_')
+ """
def pageReachabilityToken = "expr_zonemap_pruning_page_reachability_" +
UUID.randomUUID().toString()
- def pageReachabilityRows = sql """
+ sql """
SELECT '${pageReachabilityToken}', COUNT(*) FROM
test_expr_zonemap_page_reachability
WHERE starts_with(v, 'm_')
"""
- assertEquals(4096L, pageReachabilityRows[0][1] as long)
assertProfileCounterPositive(pageReachabilityToken,
"ExprZoneMapFilteredPages")
sql """ DROP TABLE IF EXISTS test_monotonic_zonemap """
@@ -221,37 +229,49 @@ suite("test_expr_zonemap_pruning") {
contains "pushAggOp=COUNT_ON_INDEX"
}
+ qt_inferred_prefix_range """
+ SELECT COUNT(*) FROM test_monotonic_zonemap
+ WHERE substring(v, 1, 1) = 'a'
+ """
def inferredPrefixRangeToken =
"expr_zonemap_pruning_inferred_prefix_range_" + UUID.randomUUID().toString()
- def inferredPrefixRangeRows = sql """
+ sql """
SELECT '${inferredPrefixRangeToken}', COUNT(*) FROM
test_monotonic_zonemap
WHERE substring(v, 1, 1) = 'a'
"""
- assertEquals(4096L, inferredPrefixRangeRows[0][1] as long)
assertProfileCounterPositive(inferredPrefixRangeToken, "RowsStatsFiltered")
+ qt_inferred_date_range """
+ SELECT COUNT(*) FROM test_monotonic_zonemap
+ WHERE date_trunc(dt, 'day') = '2026-07-28 00:00:00'
+ """
def inferredDateRangeToken = "expr_zonemap_pruning_inferred_date_range_" +
UUID.randomUUID().toString()
- def inferredDateRangeRows = sql """
+ sql """
SELECT '${inferredDateRangeToken}', COUNT(*) FROM
test_monotonic_zonemap
WHERE date_trunc(dt, 'day') = '2026-07-28 00:00:00'
"""
- assertEquals(4096L, inferredDateRangeRows[0][1] as long)
assertProfileCounterPositive(inferredDateRangeToken, "RowsStatsFiltered")
+ qt_inferred_date_format """
+ SELECT COUNT(*) FROM test_monotonic_zonemap
+ WHERE date_format(dt, '%Y-%m-%d') >= '2026-08-01'
+ """
def inferredDateFormatToken = "expr_zonemap_pruning_inferred_date_format_"
+ UUID.randomUUID().toString()
- def inferredDateFormatRows = sql """
+ sql """
SELECT '${inferredDateFormatToken}', COUNT(*) FROM
test_monotonic_zonemap
WHERE date_format(dt, '%Y-%m-%d') >= '2026-08-01'
"""
- assertEquals(4096L, inferredDateFormatRows[0][1] as long)
assertProfileCounterPositive(inferredDateFormatToken, "RowsStatsFiltered")
+ qt_inferred_date_format_equality """
+ SELECT COUNT(*) FROM test_monotonic_zonemap
+ WHERE date_format(dt, '%Y-%m-%d') = '2026-07-28'
+ """
def inferredDateFormatEqualityToken =
"expr_zonemap_pruning_inferred_date_format_equality_" +
UUID.randomUUID().toString()
- def inferredDateFormatEqualityRows = sql """
+ sql """
SELECT '${inferredDateFormatEqualityToken}', COUNT(*) FROM
test_monotonic_zonemap
WHERE date_format(dt, '%Y-%m-%d') = '2026-07-28'
"""
- assertEquals(4096L, inferredDateFormatEqualityRows[0][1] as long)
assertProfileCounterPositive(inferredDateFormatEqualityToken,
"RowsStatsFiltered")
sql """ set enable_count_on_index_pushdown = false """
@@ -262,20 +282,22 @@ suite("test_expr_zonemap_pruning") {
"""
contains "pushAggOp=NONE"
}
+ qt_inferred_direct_scan """
+ SELECT COUNT(*) FROM test_monotonic_zonemap
+ WHERE date_trunc(dt, 'day') = '2026-07-28 00:00:00'
+ """
def inferredDirectScanToken =
"expr_zonemap_pruning_inferred_direct_scan_" +
UUID.randomUUID().toString()
- def inferredDirectScanRows = sql """
+ sql """
SELECT '${inferredDirectScanToken}', COUNT(*) FROM
test_monotonic_zonemap
WHERE date_trunc(dt, 'day') = '2026-07-28 00:00:00'
"""
- assertEquals(4096L, inferredDirectScanRows[0][1] as long)
assertProfileCounterPositive(inferredDirectScanToken, "RowsStatsFiltered")
- def largeQuarterPeriodRows = sql """
+ qt_large_quarter_period """
SELECT COUNT(*) FROM test_monotonic_zonemap
WHERE quarter_floor(dt, 1431655766) = '0001-01-01 00:00:00'
"""
- assertEquals(8192L, largeQuarterPeriodRows[0][0] as long)
sql """ set enable_count_on_index_pushdown = true """
sql """ set enable_expr_zonemap_filter = true """
@@ -299,12 +321,15 @@ suite("test_expr_zonemap_pruning") {
"""
sql """ sync """
+ qt_char_pruned """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_char
+ WHERE starts_with(c, 'z') OR c IN ('z0', 'z1') OR c = 'z2'
+ """
def charToken = "expr_zonemap_pruning_char_" + UUID.randomUUID().toString()
- def charPrunedRows = sql """
+ sql """
SELECT '${charToken}', COUNT(*) FROM test_expr_zonemap_pruning_char
WHERE starts_with(c, 'z') OR c IN ('z0', 'z1') OR c = 'z2'
"""
- assertEquals(0L, charPrunedRows[0][1] as long)
assertExprZonemapPruned(charToken)
sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_nulls """
@@ -323,11 +348,410 @@ suite("test_expr_zonemap_pruning") {
sql """ INSERT INTO test_expr_zonemap_pruning_nulls VALUES (1, NULL), (2,
NULL) """
sql """ sync """
+ qt_is_not_null_pruned """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_nulls
+ WHERE v IS NOT NULL OR starts_with(v, 'zz')
+ """
def isNotNullToken = "expr_zonemap_pruning_is_not_null_" +
UUID.randomUUID().toString()
- def isNotNullPrunedRows = sql """
+ sql """
SELECT '${isNotNullToken}', COUNT(*) FROM
test_expr_zonemap_pruning_nulls
WHERE v IS NOT NULL OR starts_with(v, 'zz')
"""
- assertEquals(0L, isNotNullPrunedRows[0][1] as long)
assertExprZonemapPruned(isNotNullToken)
+
+ // Column-vs-column comparisons. A predicate over two columns of the same
table never became a
+ // ColumnPredicate, so it reaches the scanner as a common expression and
is evaluated against the
+ // segment zone map of both slots at once.
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_two_columns """
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_two_columns (
+ id INT,
+ lo INT,
+ hi INT,
+ alt INT,
+ expected INT,
+ actual INT
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ // lo lands in [0, 4095] and hi in [10000, 14095], so the two ranges are
fully separated. alt
+ // lands in [0, 7095] and equals lo on even rows and lo + 3000 on odd
ones, so lo vs alt cannot
+ // be decided from the bounds and every row has to be evaluated. expected
and actual are both
+ // the constant 7.
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_two_columns
+ SELECT CAST(number AS INT),
+ CAST(number AS INT),
+ CAST(number + 10000 AS INT),
+ IF(number % 2 = 0, CAST(number AS INT), CAST(number + 3000 AS
INT)),
+ 7,
+ 7
+ FROM numbers("number" = "4096")
+ """
+ sql """ sync """
+
+ // Runs the same query with expr zonemap pruning on and off and asserts
the two agree. The
+ // counter only shows that pruning fired; this is what shows it fired
correctly.
+ def assertSameWithAndWithoutPruning = { String predicate ->
+ sql """ set enable_expr_zonemap_filter = false """
+ def withoutPruning = sql """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_two_columns WHERE
${predicate}
+ """
+ sql """ set enable_expr_zonemap_filter = true """
+ def withPruning = sql """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_two_columns WHERE
${predicate}
+ """
+ assertEquals(withoutPruning[0][0] as long, withPruning[0][0] as long)
+ return withPruning[0][0] as long
+ }
+
+ def assertTwoColumnPruned = { String predicate, String label ->
+ def token = "expr_zonemap_pruning_two_columns_" + label + "_" +
UUID.randomUUID().toString()
+ sql """
+ SELECT '${token}', COUNT(*) FROM
test_expr_zonemap_pruning_two_columns
+ WHERE ${predicate}
+ """
+ assertExprZonemapPruned(token)
+ assertSameWithAndWithoutPruning(predicate)
+ }
+
+ // lo > hi and lo >= hi: rejected because min(hi) is already above max(lo).
+ assertTwoColumnPruned("lo > hi", "gt")
+ qt_two_column_gt """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo > hi"""
+ assertTwoColumnPruned("lo >= hi", "ge")
+ qt_two_column_ge """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo >= hi"""
+ // hi < lo and hi <= lo: the mirrored rules.
+ assertTwoColumnPruned("hi < lo", "lt")
+ qt_two_column_lt """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE hi < lo"""
+ assertTwoColumnPruned("hi <= lo", "le")
+ qt_two_column_le """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE hi <= lo"""
+ // lo = hi: the ranges are disjoint, so no row can be equal.
+ assertTwoColumnPruned("lo = hi", "eq")
+ qt_two_column_eq """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo = hi"""
+ // expected != actual: both columns collapse to the single value 7, which
is the only shape that
+ // lets != prune.
+ assertTwoColumnPruned("expected != actual", "ne")
+ qt_two_column_ne """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE expected != actual"""
+
+ // Overlapping ranges must survive, and the row counts must be exact.
These are the cases that
+ // catch a rule reading the wrong end of a range: lo in [0, 4095] against
alt in [0, 7095] cannot
+ // be separated by the bounds, so all of these have to fall through to
per-row evaluation. The
+ // enabled-vs-disabled equality stays programmatic; the exact counts land
in golden output.
+ assertSameWithAndWithoutPruning("lo < alt")
+ qt_overlap_lo_lt_alt """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo < alt"""
+ assertSameWithAndWithoutPruning("lo != alt")
+ qt_overlap_lo_ne_alt """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo != alt"""
+ assertSameWithAndWithoutPruning("lo = alt")
+ qt_overlap_lo_eq_alt """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo = alt"""
+ assertSameWithAndWithoutPruning("lo <= alt")
+ qt_overlap_lo_le_alt """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo <= alt"""
+ assertSameWithAndWithoutPruning("lo > alt")
+ qt_overlap_lo_gt_alt """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo > alt"""
+ assertSameWithAndWithoutPruning("lo >= alt - 3000")
+ qt_overlap_lo_ge_alt_shift """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo >= alt - 3000"""
+ // A cast on either side is rejected by the capability gate, so this must
still return the right
+ // answer rather than being pruned on raw bounds.
+ assertSameWithAndWithoutPruning("lo < CAST(hi AS BIGINT)")
+ qt_overlap_lo_lt_cast_hi """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo < CAST(hi AS BIGINT)"""
+
+ // One side partially NULL. min/max summarize the non-null values only,
and a NULL row makes the
+ // comparison NULL, which never satisfies the conjunct, so the separated
ranges still prune.
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_two_columns_nulls
"""
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_two_columns_nulls (
+ id INT,
+ lo INT,
+ hi INT
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_two_columns_nulls
+ SELECT CAST(number AS INT),
+ IF(number % 8 = 0, NULL, CAST(number AS INT)),
+ CAST(number + 10000 AS INT)
+ FROM numbers("number" = "4096")
+ """
+ sql """ sync """
+
+ qt_two_column_null """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_two_columns_nulls
+ WHERE lo > hi
+ """
+ def twoColumnNullToken =
+ "expr_zonemap_pruning_two_columns_null_" +
UUID.randomUUID().toString()
+ sql """
+ SELECT '${twoColumnNullToken}', COUNT(*) FROM
test_expr_zonemap_pruning_two_columns_nulls
+ WHERE lo > hi
+ """
+ assertExprZonemapPruned(twoColumnNullToken)
+
+ // A column with no non-null value at all makes the comparison NULL on
every row.
+ sql """ DROP TABLE IF EXISTS
test_expr_zonemap_pruning_two_columns_all_null """
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_two_columns_all_null (
+ id INT,
+ lo INT,
+ hi INT
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_two_columns_all_null
+ SELECT CAST(number AS INT), NULL, CAST(number AS INT)
+ FROM numbers("number" = "4096")
+ """
+ sql """ sync """
+
+ qt_two_column_all_null """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_two_columns_all_null
+ WHERE lo < hi
+ """
+ def allNullToken =
+ "expr_zonemap_pruning_two_columns_all_null_" +
UUID.randomUUID().toString()
+ sql """
+ SELECT '${allNullToken}', COUNT(*) FROM
test_expr_zonemap_pruning_two_columns_all_null
+ WHERE lo < hi
+ """
+ assertExprZonemapPruned(allNullToken)
+
+ // Same as assertExprZonemapPruned but returns the count, so a case can
pin the exact number of
+ // segments that had to be dropped instead of only that pruning happened
at all.
+ def filteredSegmentsOf = { String token ->
+ long filteredSegments = 0
+ for (int retry = 0; retry < 20; ++retry) {
+ String profile = getProfileByToken(token).toString()
+ filteredSegments = counterSum(profile,
"ExprZoneMapFilteredSegments")
+ if (filteredSegments > 0) {
+ return filteredSegments
+ }
+ Thread.sleep(500)
+ }
+ return filteredSegments
+ }
+
+ // Three loads, one segment each, each segment holding a constant in both
columns. This is the
+ // shape where a cross-column rule has to decide per segment rather than
per table, so the exact
+ // number of dropped segments is the assertion that matters.
+ //
+ // batch | a | b | a != b | a = b
+ // 1 | a | a | both collapse | overlap, keep
+ // 2 | a | b | disjoint,keep | disjoint, drop
+ // 3 | b | b | both collapse | overlap, keep
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_per_segment """
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_per_segment (
+ id INT,
+ a VARCHAR(32),
+ b VARCHAR(32)
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_per_segment
+ SELECT CAST(number AS INT), 'a', 'a' FROM numbers("number" = "1024")
+ """
+ sql """ sync """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_per_segment
+ SELECT CAST(number AS INT), 'a', 'b' FROM numbers("number" = "1024")
+ """
+ sql """ sync """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_per_segment
+ SELECT CAST(number AS INT), 'b', 'b' FROM numbers("number" = "1024")
+ """
+ sql """ sync """
+
+ qt_per_segment_ne """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_per_segment WHERE a != b
+ """
+ def perSegmentNeToken = "expr_zonemap_pruning_per_segment_ne_" +
UUID.randomUUID().toString()
+ sql """
+ SELECT '${perSegmentNeToken}', COUNT(*) FROM
test_expr_zonemap_pruning_per_segment
+ WHERE a != b
+ """
+ assertEquals(2L, filteredSegmentsOf(perSegmentNeToken))
+
+ qt_per_segment_eq """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_per_segment WHERE a = b
+ """
+ def perSegmentEqToken = "expr_zonemap_pruning_per_segment_eq_" +
UUID.randomUUID().toString()
+ sql """
+ SELECT '${perSegmentEqToken}', COUNT(*) FROM
test_expr_zonemap_pruning_per_segment
+ WHERE a = b
+ """
+ assertEquals(1L, filteredSegmentsOf(perSegmentEqToken))
+
+ sql """ set enable_expr_zonemap_filter = false """
+ qt_per_segment_ne_without """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_per_segment WHERE a != b
+ """
+ qt_per_segment_eq_without """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_per_segment WHERE a = b
+ """
+ sql """ set enable_expr_zonemap_filter = true """
+
+ // The single-slot path on the same table, to show that widening the
zonemap capability gate for
+ // two-slot shapes did not disturb the one-slot one it shares a gate with.
It has to be a
+ // predicate that expression zone maps actually see: `a = 'zzz'` is turned
into a ColumnPredicate
+ // and pruned by the ordinary olap path instead, which leaves
ExprZoneMapFilteredSegments at 0.
+ qt_literal_still_prunes """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_per_segment WHERE
starts_with(a, 'z')
+ """
+ def literalStillPrunesToken =
+ "expr_zonemap_pruning_per_segment_literal_" +
UUID.randomUUID().toString()
+ sql """
+ SELECT '${literalStillPrunesToken}', COUNT(*) FROM
test_expr_zonemap_pruning_per_segment
+ WHERE starts_with(a, 'z')
+ """
+ assertEquals(3L, filteredSegmentsOf(literalStillPrunesToken))
+
+ // And the ColumnPredicate shape still returns the right answer.
+ qt_column_predicate """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_per_segment WHERE a =
'zzz'
+ """
+
+ // Two DOUBLE columns on a native table. Unlike Parquet, the segment
writer records NaN presence
+ // in the zone map, so a float column with no NaN in it prunes normally.
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_two_columns_double
"""
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_two_columns_double (
+ id INT,
+ lo DOUBLE,
+ hi DOUBLE
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_two_columns_double
+ SELECT CAST(number AS INT), CAST(number AS DOUBLE), CAST(number +
10000 AS DOUBLE)
+ FROM numbers("number" = "4096")
+ """
+ sql """ sync """
+
+ qt_double_pruned """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_two_columns_double
+ WHERE lo > hi
+ """
+ def doubleToken = "expr_zonemap_pruning_two_columns_double_" +
UUID.randomUUID().toString()
+ sql """
+ SELECT '${doubleToken}', COUNT(*) FROM
test_expr_zonemap_pruning_two_columns_double
+ WHERE lo > hi
+ """
+ assertExprZonemapPruned(doubleToken)
+
+ // NaN and infinities make the bounds unusable for range pruning, so these
cases only have to
+ // return the right rows. has_nan / has_positive_inf / has_negative_inf
are what stops them.
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_two_columns_nan """
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_two_columns_nan (
+ id INT,
+ lo DOUBLE,
+ hi DOUBLE
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_two_columns_nan VALUES
+ (1, CAST('NaN' AS DOUBLE), 1.0),
+ (2, 1.0, CAST('NaN' AS DOUBLE)),
+ (3, CAST('infinity' AS DOUBLE), 1.0),
+ (4, CAST('-infinity' AS DOUBLE), 1.0),
+ (5, 1.0, 2.0),
+ (6, 2.0, 1.0)
+ """
+ sql """ sync """
+
+ def assertSameOnNanTable = { String predicate ->
+ sql """ set enable_expr_zonemap_filter = false """
+ def without = sql """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_two_columns_nan
WHERE ${predicate}
+ """
+ sql """ set enable_expr_zonemap_filter = true """
+ def with_ = sql """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_two_columns_nan
WHERE ${predicate}
+ """
+ assertEquals(without[0][0] as long, with_[0][0] as long)
+ return with_[0][0] as long
+ }
+
+ // Doris orders NaN above every other value and treats NaN = NaN as true.
So lo > hi holds for
+ // rows 1, 3 and 6; lo < hi for rows 2, 4 and 5; and no row has the two
columns equal. The
+ // enabled-vs-disabled equality stays programmatic; the exact counts land
in golden output.
+ assertSameOnNanTable("lo > hi")
+ qt_nan_lo_gt_hi """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns_nan WHERE lo > hi"""
+ assertSameOnNanTable("lo != hi")
+ qt_nan_lo_ne_hi """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns_nan WHERE lo != hi"""
+ assertSameOnNanTable("lo = hi")
+ qt_nan_lo_eq_hi """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns_nan WHERE lo = hi"""
+ assertSameOnNanTable("lo < hi")
+ qt_nan_lo_lt_hi """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns_nan WHERE lo < hi"""
+
+ // __DORIS_VERSION_COL__ stores 0 on disk; the 4.2 reader substitutes the
rowset version after
+ // reading a single-version segment. Expression zone-map pruning must not
use the on-disk
+ // [0,0] placeholder at either segment or page level.
+ sql """ set show_hidden_columns = true """
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_hidden_col """
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_hidden_col (
+ k INT,
+ v INT
+ ) UNIQUE KEY(k)
+ DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES (
+ "enable_unique_key_merge_on_write" = "true",
+ "replication_num" = "1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sql """ INSERT INTO test_expr_zonemap_pruning_hidden_col SELECT number,
number FROM numbers("number" = "2048") """
+ sql """ sync """
+ def hiddenVersionRows = sql """ SELECT DISTINCT __DORIS_VERSION_COL__ FROM
test_expr_zonemap_pruning_hidden_col """
+ assertEquals(1, hiddenVersionRows.size())
+ long hiddenVersion = hiddenVersionRows[0][0] as long
+ assertTrue(hiddenVersion > 0)
+ def assertSameOnHiddenCol = { String predicate ->
+ sql """ set enable_expr_zonemap_filter = false """
+ def without = sql """ SELECT COUNT(*) FROM
test_expr_zonemap_pruning_hidden_col WHERE ${predicate} """
+ sql """ set enable_expr_zonemap_filter = true """
+ def with_ = sql """ SELECT COUNT(*) FROM
test_expr_zonemap_pruning_hidden_col WHERE ${predicate} """
+ assertEquals(without[0][0] as long, with_[0][0] as long)
+ return with_[0][0] as long
+ }
+ // Every row carries the version and `v = -1` matches nothing, so the
compound is the full 2048
+ // rows. Pruning against the [0,0] placeholder would drop the segment.
+ assertSameOnHiddenCol("__DORIS_VERSION_COL__ = ${hiddenVersion} OR v = -1")
+ qt_hidden_version_or_miss """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_hidden_col WHERE __DORIS_VERSION_COL__ =
${hiddenVersion} OR v = -1"""
+ sql """ set show_hidden_columns = false """
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
