This is an automated email from the ASF dual-hosted git repository.
csun5285 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 349cd15471d [Enhancement](zonemap) Support column-vs-column
comparisons in expression zone map pruning (#67774)
349cd15471d is described below
commit 349cd15471d7531a687f71db19d5742b3759d4cc
Author: YangJie <[email protected]>
AuthorDate: Tue Oct 20 02:39:50 2026 -0400
[Enhancement](zonemap) Support column-vs-column comparisons in expression
zone map pruning (#67774)
Zone map pruning only fires when a predicate compares a column against a
constant. A predicate over two columns of the same table, `WHERE a != b`
or `WHERE a < b`, never participates, so every segment and every Parquet
row group is read even when the two columns' min/max ranges make the
predicate unsatisfiable.
Everything on the path except the leaf already supports this.
`ZoneMapEvalContext` is keyed per slot and the segment-level and Parquet
row-group-level callers already populate an entry for every slot a
compound expression references, which is why cross-column AND / OR
compounds prune today. What was missing is the comparison operator's
capability gate: it accepts one slot plus one literal and nothing else,
so a two-slot comparison is dropped before evaluation.
This PR adds the slot-vs-slot shape next to the existing slot-vs-literal
one and gives the six always-false tests a single implementation shared
by both. With the left operand's non-null values in `[lmin, lmax]` and
the right operand's in `[rmin, rmax]`, the comparison cannot be TRUE for
any row when:
| operator | condition |
| --- | --- |
| `=` | `lmin > rmax \|\| rmin > lmax` |
| `!=` | `lmin == lmax && rmin == rmax && lmin == rmin` |
| `<` | `lmin >= rmax` |
| `<=` | `lmin > rmax` |
| `>` | `rmin >= lmax` |
| `>=` | `rmin > lmax` |
Substituting the degenerate range `[literal, literal]` for the right
operand reduces each row to the slot-vs-literal test already in the
tree, so the single-column path keeps its exact current behaviour. The
existing tests passing unchanged is the mechanical check on that.
`kNoMatch` means "no row in this zone can make the conjunct TRUE", not
"every row is FALSE", so NULL does not weaken these rules: a WHERE
conjunct discards FALSE and NULL alike, and the bounds summarize exactly
the non-null values. A column with no non-null value at all makes the
comparison NULL on every row and returns `kNoMatch` unconditionally.
One trap for reviewers: `can_evaluate` must not be widened. It is shared
by dictionary filtering and by `can_evaluate_equality`, and both
dereference `extract_slot_and_literal` behind a `DORIS_CHECK`, so
accepting a slot-vs-slot expression there would abort. This PR adds a
separate `can_evaluate_slot_slot` and ORs it into
`can_evaluate_zonemap_filter` only; a unit test pins that down.
Three deliberate limitations: a cast around either operand is rejected,
because the bounds are raw stored values and the optimizer inserts a
cast whenever the two column types differ; on Parquet a floating column
pair does not prune at all, because Parquet bounds omit NaN without
reporting how many were skipped, which would flip the `!=` rule; and
VARIANT, JSONB, ARRAY/MAP/STRUCT and aggregate-state columns are not
admitted, because `Field` compares any two of their values as equal, and
a segment types a VARIANT path slot by its inferred leaf type, which the
expression's VARIANT type would not match.
The v1 Parquet reader is excluded from slot-vs-slot pruning entirely,
because that reader is being removed. `_process_expr_zonemap_filter`
restricts itself to single-slot conjuncts (the v1 page path already
required a single column), so column-vs-column pruning runs on native
segments and the v2 Parquet reader only. This also moots the v1-specific
slot-vs-slot review threads here (malformed fixed-width statistics,
VARBINARY mapping, and the adjusted-UTC timestamp rollback tracked by
the now-closed #68118). The v1 single-column path keeps the
decimal-domain and NaN fixes described next.
A limitation on the v1 single-column path came out of review.
`ParquetPredicate::parse_min_max_value` decodes v1 bounds in the file's
own logical type, while `_type_matches` compares only primitive types,
so a file DECIMAL at scale 3 and a table DECIMAL at scale 2 both passed
and their unscaled payloads were then compared as if they shared a
domain. The v1 reader now leaves the zone map out unless the file and
table types agree, which is the check `format_v2` already applies. That
costs pruning in three narrow places, all of them correctness-driven: a
DATETIMEV2 or TIMESTAMPTZ column whose declared scale differs from the
Parquet unit loses expression zone map pruning entirely, which is the
widest practical case and matches the policy already written down for
the slot-vs-literal path; a decimal pair differing only in precision is
domain-identical yet also skipped; and IS NULL / IS NOT NULL pruning
goes with it, because a zone map carrying only the null flags leaves
min/max at `TYPE_NULL` and any range predicate would then fail
`range_stats_usable_for_zonemap`'s type check.
This also fixes a bug it uncovered. The flag that carries "these bounds
came from Parquet, so a hidden NaN is possible" was only ever set in the
format_v2 reader; the v1 reader assigned the slot's data type and left
the flag at its default, so the NaN guard never fired on that path. Two
DOUBLE columns whose Parquet bounds both read `[1.0, 1.0]` were then
pruned by the `!=` rule even when one of them hid a NaN, dropping a row
that should have come back. The rule now lives in one setter on
`SlotZoneMap` that assigns the type and the flag together, and all three
Parquet call sites go through it.
The hidden version and TSO columns need no special handling here: since
#65684, a single-version segment reads them through a
`ConstantColumnReader` whose zone map holds the substituted value, so
pruning sees the same value the rows come back with.
---
be/src/exprs/expr_zonemap_filter.cpp | 33 ++
be/src/exprs/expr_zonemap_filter.h | 25 +
be/src/exprs/function/functions_comparison.h | 186 ++++++--
be/src/format/parquet/vparquet_reader.cpp | 41 +-
be/src/format_v2/parquet/parquet_statistics.cpp | 5 +-
.../storage/index/zone_map/zonemap_eval_context.h | 17 +
be/test/exprs/expr_zonemap_filter_test.cpp | 465 ++++++++++++++++++
.../expr_zonemap/test_expr_zonemap_pruning.out | 132 ++++++
.../expr_zonemap/test_expr_zonemap_pruning.groovy | 519 +++++++++++++++++++--
9 files changed, 1342 insertions(+), 81 deletions(-)
diff --git a/be/src/exprs/expr_zonemap_filter.cpp
b/be/src/exprs/expr_zonemap_filter.cpp
index 339123c7aad..a144a9a0768 100644
--- a/be/src/exprs/expr_zonemap_filter.cpp
+++ b/be/src/exprs/expr_zonemap_filter.cpp
@@ -250,6 +250,39 @@ std::optional<SlotLiteral> extract_slot_and_literal(const
VExprSPtrs& args) {
return std::nullopt;
}
+std::optional<SlotSlot> extract_slot_and_slot(const VExprSPtrs& args) {
+ if (args.size() != 2) {
+ return std::nullopt;
+ }
+ auto left = std::dynamic_pointer_cast<VSlotRef>(args[0]);
+ auto right = std::dynamic_pointer_cast<VSlotRef>(args[1]);
+ if (left == nullptr || right == nullptr) {
+ return std::nullopt;
+ }
+ return SlotSlot {.left_slot_index = left->column_id(),
+ .left_type = left->data_type(),
+ .right_slot_index = right->column_id(),
+ .right_type = right->data_type()};
+}
+
+bool contains_slot_slot_comparison(const VExprSPtr& expr) {
+ if (expr == nullptr) {
+ return false;
+ }
+ // A comparison node's two operands are its children, so a slot-vs-slot
leaf is a node whose
+ // children are two VSlotRef. extract_slot_and_slot accepts a same-column
pair too, which is the
+ // point: `a < a` has one column id but is still slot-vs-slot.
+ if (extract_slot_and_slot(expr->children()).has_value()) {
+ return true;
+ }
+ for (const auto& child : expr->children()) {
+ if (contains_slot_slot_comparison(child)) {
+ return true;
+ }
+ }
+ return false;
+}
+
std::optional<SlotLiteral> extract_array_contains_slot_and_literal(const
VExprSPtrs& args) {
if (args.size() != 2) {
return std::nullopt;
diff --git a/be/src/exprs/expr_zonemap_filter.h
b/be/src/exprs/expr_zonemap_filter.h
index 311d0fb8c65..ada45307b51 100644
--- a/be/src/exprs/expr_zonemap_filter.h
+++ b/be/src/exprs/expr_zonemap_filter.h
@@ -97,6 +97,21 @@ struct SlotLiteral {
bool literal_on_left;
};
+// A comparison whose both operands are plain slot references, e.g. `WHERE a
!= b`. Unlike
+// SlotLiteral there is no flag to flip the operator: with two slots the
argument order already is
+// the expression order, so evaluators use the comparison operator as written.
+struct SlotSlot {
+ // Slot ordinals of the left and right operand. Both are keys used to look
up the corresponding
+ // reader-schema type and zone map from ZoneMapEvalContext, same as
SlotLiteral::slot_index.
+ int left_slot_index;
+ // Type carried by the left slot expression. Kept separately for the same
reason as
+ // SlotLiteral::slot_type: evaluation first has to verify the expression
binding is compatible
+ // with the reader-schema type held by the context.
+ DataTypePtr left_type;
+ int right_slot_index;
+ DataTypePtr right_type;
+};
+
enum class MetadataPathKind {
STRUCT_FIELD,
LIST_ELEMENT,
@@ -120,6 +135,16 @@ struct MetadataProbe {
std::optional<SlotLiteral> extract_slot_and_literal(const VExprSPtrs& args);
+// Both operands must be bare VSlotRef. A cast around a slot is deliberately
rejected: the zone map
+// holds raw stored values, so comparing across a cast would need the cast
applied to the bounds.
+std::optional<SlotSlot> extract_slot_and_slot(const VExprSPtrs& args);
+
+// True if the expression tree contains a slot-vs-slot comparison leaf (two
VSlotRef operands),
+// at the root or nested under AND/OR, including two references to the same
column (`a < a`). The v1
+// Parquet reader uses this to exclude slot-vs-slot pruning entirely: counting
distinct columns is
+// not enough because a same-column pair still has one column id.
+bool contains_slot_slot_comparison(const VExprSPtr& expr);
+
std::optional<SlotLiteral> extract_array_contains_slot_and_literal(const
VExprSPtrs& args);
ZoneMapFilterResult evaluate_array_contains_zonemap(const ZoneMapEvalContext&
ctx,
diff --git a/be/src/exprs/function/functions_comparison.h
b/be/src/exprs/function/functions_comparison.h
index 0f21c80c140..3b2146f28dd 100644
--- a/be/src/exprs/function/functions_comparison.h
+++ b/be/src/exprs/function/functions_comparison.h
@@ -50,6 +50,7 @@
#include "exprs/vexpr.h"
#include "storage/index/index_reader_helper.h"
#include "storage/index/inverted/inverted_index_iterator.h"
+#include "storage/olap_define.h"
namespace doris {
/** Comparison functions: ==, !=, <, >, <=, >=.
@@ -293,16 +294,43 @@ inline Op symmetric_op(Op op) {
__builtin_unreachable();
}
-inline ZoneMapFilterResult evaluate(const ZoneMapEvalContext& ctx, const
VExprSPtrs& arguments,
- Op op) {
- auto slot_literal = expr_zonemap::extract_slot_and_literal(arguments);
+// Return true when `L <op> R` cannot be TRUE for any row, given that every
non-null value of the
+// left operand lies in [lmin, lmax] and every non-null value of the right
operand lies in
+// [rmin, rmax]. This only compares the four bounds: callers must already have
rejected zone maps
+// whose bounds are unusable and columns that hold no non-null value at all.
+//
+// The slot-vs-literal path passes the literal as the degenerate range
[literal, literal], which
+// reduces every branch below to the plain slot-vs-literal test it replaced.
+inline bool range_vs_range_no_match(const Field& lmin, const Field& lmax,
const Field& rmin,
+ const Field& rmax, Op op) {
+ switch (op) {
+ case Op::EQ:
+ // Disjoint ranges cannot share a value.
+ return lmin > rmax || rmin > lmax;
+ case Op::NE:
+ // Only provable when both sides collapse to the same single value.
+ return lmin == lmax && rmin == rmax && lmin == rmin;
+ case Op::LT:
+ return lmin >= rmax;
+ case Op::LE:
+ return lmin > rmax;
+ case Op::GT:
+ return rmin >= lmax;
+ case Op::GE:
+ return rmin > lmax;
+ }
+ __builtin_unreachable();
+}
- auto slot_type = expr_zonemap::fetch_compatible_slot_type(ctx,
slot_literal->slot_index,
-
slot_literal->slot_type);
+inline ZoneMapFilterResult evaluate_slot_literal(const ZoneMapEvalContext& ctx,
+ const
expr_zonemap::SlotLiteral& slot_literal,
+ Op op) {
+ auto slot_type = expr_zonemap::fetch_compatible_slot_type(ctx,
slot_literal.slot_index,
+
slot_literal.slot_type);
if (slot_type == nullptr) {
return unsupported_zonemap_filter(ctx);
}
- auto zone_map_ptr = ctx.zone_map(slot_literal->slot_index);
+ auto zone_map_ptr = ctx.zone_map(slot_literal.slot_index);
if (zone_map_ptr == nullptr) {
return unsupported_zonemap_filter(ctx);
}
@@ -314,42 +342,95 @@ inline ZoneMapFilterResult evaluate(const
ZoneMapEvalContext& ctx, const VExprSP
return unsupported_zonemap_filter(ctx);
}
- const auto effective_op = slot_literal->literal_on_left ? symmetric_op(op)
: op;
- const auto& literal = slot_literal->literal;
+ const auto effective_op = slot_literal.literal_on_left ? symmetric_op(op)
: op;
+ const auto& literal = slot_literal.literal;
const bool literal_is_nan = literal.is_nan();
const bool hidden_nan_can_match = (effective_op == Op::EQ &&
literal_is_nan) ||
(effective_op == Op::NE &&
!literal_is_nan) ||
(effective_op == Op::GT &&
!literal_is_nan) ||
effective_op == Op::GE;
- if (ctx.floating_nan_count_unknown(slot_literal->slot_index) &&
hidden_nan_can_match) {
+ if (ctx.floating_nan_count_unknown(slot_literal.slot_index) &&
hidden_nan_can_match) {
// Parquet bounds omit NaNs, so only operators that cannot match a
hidden NaN may prune.
return unsupported_zonemap_filter(ctx);
}
- switch (effective_op) {
- case Op::EQ:
- return literal < zone_map.min_value || zone_map.max_value < literal
- ? ZoneMapFilterResult::kNoMatch
- : ZoneMapFilterResult::kMayMatch;
- case Op::NE:
- return zone_map.min_value == literal && zone_map.max_value == literal
- ? ZoneMapFilterResult::kNoMatch
- : ZoneMapFilterResult::kMayMatch;
- case Op::LT:
- return zone_map.min_value >= literal ? ZoneMapFilterResult::kNoMatch
- : ZoneMapFilterResult::kMayMatch;
- case Op::LE:
- return zone_map.min_value > literal ? ZoneMapFilterResult::kNoMatch
- : ZoneMapFilterResult::kMayMatch;
- case Op::GT:
- return zone_map.max_value <= literal ? ZoneMapFilterResult::kNoMatch
- : ZoneMapFilterResult::kMayMatch;
- case Op::GE:
- return zone_map.max_value < literal ? ZoneMapFilterResult::kNoMatch
- : ZoneMapFilterResult::kMayMatch;
+ return range_vs_range_no_match(zone_map.min_value, zone_map.max_value,
literal, literal,
+ effective_op)
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+}
+
+inline ZoneMapFilterResult evaluate_slot_slot(const ZoneMapEvalContext& ctx,
+ const expr_zonemap::SlotSlot&
slot_slot, Op op) {
+ const auto left_type = expr_zonemap::fetch_compatible_slot_type(ctx,
slot_slot.left_slot_index,
+
slot_slot.left_type);
+ const auto right_type = expr_zonemap::fetch_compatible_slot_type(
+ ctx, slot_slot.right_slot_index, slot_slot.right_type);
+ if (left_type == nullptr || right_type == nullptr) {
+ // The context skips a slot entirely when the segment cannot apply
predicates on it.
+ return unsupported_zonemap_filter(ctx);
}
+ const auto left_zone_map = ctx.zone_map(slot_slot.left_slot_index);
+ const auto right_zone_map = ctx.zone_map(slot_slot.right_slot_index);
+ if (left_zone_map == nullptr || right_zone_map == nullptr) {
+ // A slot can be present with a data type but no zone map, so this is
a separate check.
+ return unsupported_zonemap_filter(ctx);
+ }
+ // A column holding no non-null value makes the comparison NULL on every
row, which never
+ // satisfies a WHERE conjunct. This must run before the range checks
below: an all-null zone map
+ // leaves min/max default-constructed as TYPE_NULL, which the range checks
would fatal on.
+ if (!left_zone_map->has_not_null || !right_zone_map->has_not_null) {
+ return ZoneMapFilterResult::kNoMatch;
+ }
+ if (!expr_zonemap::range_stats_usable_for_zonemap(*left_zone_map,
left_type) ||
+ !expr_zonemap::range_stats_usable_for_zonemap(*right_zone_map,
right_type)) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ // A native string zone-map max of exactly MAX_ZONE_MAP_INDEX_SIZE bytes
was bumped upward,
+ // and may also have been truncated (segments written before the carry fix
may hold a bump
+ // that wrapped on 0xff), so a bound of that length is not a reliable
fence for a two-sided
+ // proof. A shorter bound is the exact value; the bump never changes the
length, so this also
+ // rejects those wrapped bounds.
+ // The test only decides whether to trust the bound as a fence. A
Parquet-sourced string bound
+ // shorter than the cap is not necessarily exact, but it is a
spec-conservative min/max, which
+ // is all a range proof needs; do not tighten this to assume exactness
below the cap.
+ // data_types_compatible pairs strings only with strings, so checking one
side's type is enough
+ // to know all four bounds are strings.
+ if (is_string_type(remove_nullable(left_type)->get_primitive_type())) {
+ auto is_untruncated = [](const Field& f) {
+ return f.get<TYPE_STRING>().size() < MAX_ZONE_MAP_INDEX_SIZE;
+ };
+ if (!is_untruncated(left_zone_map->min_value) ||
+ !is_untruncated(left_zone_map->max_value) ||
+ !is_untruncated(right_zone_map->min_value) ||
+ !is_untruncated(right_zone_map->max_value)) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ }
+ // Parquet bounds omit NaN without recording how many were skipped, so a
zone map that looks
+ // like a single point may still hide one, which would flip the NE rule
from false to true. The
+ // slot-vs-literal path can reason per operator because it knows whether
the literal is NaN;
+ // with two slots there is no literal, so bail out for every operator
instead.
+ if (ctx.floating_nan_count_unknown(slot_slot.left_slot_index) ||
+ ctx.floating_nan_count_unknown(slot_slot.right_slot_index)) {
+ return unsupported_zonemap_filter(ctx);
+ }
+ return range_vs_range_no_match(left_zone_map->min_value,
left_zone_map->max_value,
+ right_zone_map->min_value,
right_zone_map->max_value, op)
+ ? ZoneMapFilterResult::kNoMatch
+ : ZoneMapFilterResult::kMayMatch;
+}
- // keep this to avoid compile failure with g++.
- __builtin_unreachable();
+inline ZoneMapFilterResult evaluate(const ZoneMapEvalContext& ctx, const
VExprSPtrs& arguments,
+ Op op) {
+ if (auto slot_literal = expr_zonemap::extract_slot_and_literal(arguments);
+ slot_literal.has_value()) {
+ return evaluate_slot_literal(ctx, *slot_literal, op);
+ }
+ auto slot_slot = expr_zonemap::extract_slot_and_slot(arguments);
+ // can_evaluate_zonemap_filter accepts exactly the two shapes handled
above, so reaching this
+ // point means the capability gate and this evaluator went out of sync.
+ DORIS_CHECK(slot_slot.has_value());
+ return evaluate_slot_slot(ctx, *slot_slot, op);
}
inline bool can_evaluate(const VExprSPtrs& arguments) {
@@ -377,6 +458,40 @@ inline bool can_evaluate(const VExprSPtrs& arguments) {
return true;
}
+// Accept a comparison whose both operands are slot references. Kept separate
from can_evaluate on
+// purpose: can_evaluate also gates dictionary filtering and
can_evaluate_equality, and both of
+// those dereference extract_slot_and_literal behind a DORIS_CHECK, so
widening it would abort on a
+// slot-vs-slot expression. Only can_evaluate_zonemap_filter ORs this in.
+inline bool can_evaluate_slot_slot(const VExprSPtrs& arguments) {
+ auto slot_slot = expr_zonemap::extract_slot_and_slot(arguments);
+ if (!slot_slot.has_value()) {
+ return false;
+ }
+ DORIS_CHECK(slot_slot->left_type != nullptr);
+ DORIS_CHECK(slot_slot->right_type != nullptr);
+ // Reject types whose slot-slot bounds cannot be compared safely:
+ // - VARBINARY: v1 Parquet produces a TYPE_STRING bound for it, which
+ // range_stats_usable_for_zonemap aborts on instead of falling back.
+ // - VARIANT: a segment types a path slot by its inferred leaf type (a
typed `v.a` is INT), so
+ // fetch_compatible_slot_type's check against the VARIANT expression
type would fail.
+ // - ARRAY/MAP/STRUCT, JSONB, aggregate states: Field orders every pair of
them as equal.
+ // A truncated string bound is rejected later, in evaluate_slot_slot, by
its length.
+ const auto excluded = [](const DataTypePtr& type) {
+ const auto primitive_type =
remove_nullable(type)->get_primitive_type();
+ return is_varbinary(primitive_type) || primitive_type == TYPE_VARIANT
||
+ primitive_type == TYPE_JSONB || is_complex_type(primitive_type)
||
+ is_var_len_object(primitive_type);
+ };
+ if (excluded(slot_slot->left_type) || excluded(slot_slot->right_type)) {
+ return false;
+ }
+ // The two zone maps' Fields are compared directly and Field comparison
throws on mismatched
+ // non-string types, so reject incompatible column pairs here. A pair
differing only by width or
+ // decimal scale never reaches this point anyway, because the optimizer
inserts a cast and a
+ // cast is not a VSlotRef.
+ return expr_zonemap::data_types_compatible(slot_slot->left_type,
slot_slot->right_type);
+}
+
inline bool can_evaluate_equality(const VExprSPtrs& arguments, Op op) {
if (op != Op::EQ || !can_evaluate(arguments)) {
return false;
@@ -736,8 +851,11 @@ public:
}
bool can_evaluate_zonemap_filter(const VExprSPtrs& arguments) const
override {
- return comparison_zonemap_detail::op_from_name(name).has_value() &&
- comparison_zonemap_detail::can_evaluate(arguments);
+ if (!comparison_zonemap_detail::op_from_name(name).has_value()) {
+ return false;
+ }
+ return comparison_zonemap_detail::can_evaluate(arguments) ||
+ comparison_zonemap_detail::can_evaluate_slot_slot(arguments);
}
ZoneMapFilterResult evaluate_dictionary_filter(const
DictionaryEvalContext& ctx,
diff --git a/be/src/format/parquet/vparquet_reader.cpp
b/be/src/format/parquet/vparquet_reader.cpp
index e5197896bd7..79ea2bb19f6 100644
--- a/be/src/format/parquet/vparquet_reader.cpp
+++ b/be/src/format/parquet/vparquet_reader.cpp
@@ -1138,6 +1138,11 @@ bool ParquetReader::_expr_zonemap_page_slot_index(const
VExprContextSPtr& conjun
if (column_ids.size() != 1) {
return false;
}
+ // The v1 reader does not take on slot-vs-slot pruning. A same-column pair
like `a < a` still has
+ // one column id, so exclude any slot-vs-slot leaf explicitly rather than
by column count.
+ if (expr_zonemap::contains_slot_slot_comparison(conjunct->root())) {
+ return false;
+ }
const int slot_index = *column_ids.begin();
if (slot_index < 0 || static_cast<size_t>(slot_index) >=
_tuple_descriptor->slots().size()) {
@@ -1400,6 +1405,11 @@ Status ParquetReader::_process_expr_zonemap_page_filter(
if (!cached_page_index->get_stat_func(&stat, cid) || stat == nullptr
|| !stat->available) {
continue;
}
+ // Same domain requirement as the row-group path above.
+ if (stat->col_schema == nullptr ||
+ !expr_zonemap::data_types_compatible(stat->col_schema->data_type,
slot->type())) {
+ continue;
+ }
RowRanges expr_ranges;
ZoneMapEvalStats page_stats;
for (int64_t page_id = 0; page_id < stat->num_of_pages; ++page_id) {
@@ -1411,7 +1421,7 @@ Status ParquetReader::_process_expr_zonemap_page_filter(
ZoneMapEvalContext ctx;
ZoneMapEvalContext::SlotZoneMap slot_zone_map;
- slot_zone_map.data_type = slot->type();
+ slot_zone_map.set_data_type_from_parquet(slot->type());
segment_v2::ZoneMap zone_map;
zone_map.has_null = stat->has_null[page_id];
zone_map.has_not_null = !stat->is_all_null[page_id];
@@ -1656,11 +1666,24 @@ Status
ParquetReader::_process_expr_zonemap_filter(const tparquet::RowGroup& row
return Status::OK();
}
+ // The v1 Parquet reader is being removed, so it does not take on the new
slot-vs-slot pruning.
+ // Restrict this path to single-column conjuncts that carry no
slot-vs-slot leaf: counting
+ // distinct columns alone is not enough, because a same-column pair like
`a < a` still has one
+ // column id. A two-slot shape falls back to no pruning here. Native and
v2 keep it.
std::set<int> column_ids;
+ VExprContextSPtrs single_slot_conjuncts;
for (const auto& conjunct : all_conjuncts) {
- if (conjunct->root() != nullptr &&
conjunct->root()->can_evaluate_zonemap_filter()) {
- conjunct->root()->collect_slot_column_ids(column_ids);
+ if (conjunct->root() == nullptr ||
!conjunct->root()->can_evaluate_zonemap_filter()) {
+ continue;
}
+ std::set<int> conjunct_column_ids;
+ conjunct->root()->collect_slot_column_ids(conjunct_column_ids);
+ if (conjunct_column_ids.size() != 1 ||
+ expr_zonemap::contains_slot_slot_comparison(conjunct->root())) {
+ continue;
+ }
+ single_slot_conjuncts.emplace_back(conjunct);
+ column_ids.insert(*conjunct_column_ids.begin());
}
if (column_ids.empty()) {
return Status::OK();
@@ -1673,7 +1696,7 @@ Status ParquetReader::_process_expr_zonemap_filter(const
tparquet::RowGroup& row
}
auto* slot = _tuple_descriptor->slots()[cid];
ZoneMapEvalContext::SlotZoneMap slot_zone_map;
- slot_zone_map.data_type = slot->type();
+ slot_zone_map.set_data_type_from_parquet(slot->type());
if (!_exists_in_file(slot->col_name()) || !_type_matches(cid)) {
ctx.slots.emplace(cid, std::move(slot_zone_map));
continue;
@@ -1681,6 +1704,14 @@ Status ParquetReader::_process_expr_zonemap_filter(const
tparquet::RowGroup& row
const auto& file_col_name =
_table_info_node_ptr->children_file_column_name(slot->col_name());
const FieldSchema* col_schema =
_file_metadata->schema().get_column(file_col_name);
+ // parse_min_max_value decodes the bounds in the file's own logical
type, while _type_matches
+ // only compares primitive types. A DECIMAL bound decoded at the
file's scale would then be
+ // compared against the table's scale as if the payloads shared a
domain, so leave the zone
+ // map out unless the two types agree exactly.
+ if (!expr_zonemap::data_types_compatible(col_schema->data_type,
slot->type())) {
+ ctx.slots.emplace(cid, std::move(slot_zone_map));
+ continue;
+ }
int parquet_col_id = col_schema->physical_column_index;
if (parquet_col_id < 0) {
// Complex parent fields do not map to a physical Parquet column.
@@ -1714,7 +1745,7 @@ Status ParquetReader::_process_expr_zonemap_filter(const
tparquet::RowGroup& row
ctx.slots.emplace(cid, std::move(slot_zone_map));
}
- const auto result = VExprContext::evaluate_zonemap_filter(all_conjuncts,
ctx);
+ const auto result =
VExprContext::evaluate_zonemap_filter(single_slot_conjuncts, ctx);
ctx.stats.accumulate_to(&_reader_statistics);
if (result == ZoneMapFilterResult::kNoMatch) {
*filter_group = true;
diff --git a/be/src/format_v2/parquet/parquet_statistics.cpp
b/be/src/format_v2/parquet/parquet_statistics.cpp
index 916aaa6c1c6..62dc379cd06 100644
--- a/be/src/format_v2/parquet/parquet_statistics.cpp
+++ b/be/src/format_v2/parquet/parquet_statistics.cpp
@@ -945,11 +945,8 @@ void add_slot_zonemap(ZoneMapEvalContext* ctx, int
slot_index, const DataTypePtr
std::shared_ptr<segment_v2::ZoneMap> zone_map) {
DORIS_CHECK(ctx != nullptr);
ZoneMapEvalContext::SlotZoneMap slot_zone_map;
- slot_zone_map.data_type = data_type;
+ slot_zone_map.set_data_type_from_parquet(data_type);
slot_zone_map.zone_map = std::move(zone_map);
- const auto primitive_type =
remove_nullable(data_type)->get_primitive_type();
- slot_zone_map.floating_nan_count_unknown =
- primitive_type == TYPE_FLOAT || primitive_type == TYPE_DOUBLE;
ctx->slots.emplace(slot_index, std::move(slot_zone_map));
}
diff --git a/be/src/storage/index/zone_map/zonemap_eval_context.h
b/be/src/storage/index/zone_map/zonemap_eval_context.h
index 7c1afab8780..25eb935ac44 100644
--- a/be/src/storage/index/zone_map/zonemap_eval_context.h
+++ b/be/src/storage/index/zone_map/zonemap_eval_context.h
@@ -22,6 +22,7 @@
#include <memory>
#include "core/data_type/data_type.h"
+#include "core/data_type/data_type_nullable.h"
#include "storage/index/zone_map/zone_map_index.h"
#include "storage/index/zone_map/zonemap_filter_result.h"
@@ -54,6 +55,22 @@ public:
std::shared_ptr<const segment_v2::ZoneMap> zone_map;
// Parquet min/max does not expose whether a floating chunk also
contains NaNs.
bool floating_nan_count_unknown = false;
+
+ // Every caller that fills this from Parquet statistics must go
through here. Setting
+ // data_type without the flag silently re-enables pruning that Parquet
bounds cannot
+ // support, so the two fields are assigned together on purpose.
+ void set_data_type_from_parquet(const DataTypePtr& type) {
+ data_type = type;
+ // SlotDescriptor's test-only default constructor leaves the type
null, and the callers
+ // used to just copy it through.
+ floating_nan_count_unknown = false;
+ if (type == nullptr) {
+ return;
+ }
+ const auto primitive_type =
remove_nullable(type)->get_primitive_type();
+ floating_nan_count_unknown =
+ primitive_type == TYPE_FLOAT || primitive_type ==
TYPE_DOUBLE;
+ }
};
std::shared_ptr<const segment_v2::ZoneMap> zone_map(int slot_index) const;
diff --git a/be/test/exprs/expr_zonemap_filter_test.cpp
b/be/test/exprs/expr_zonemap_filter_test.cpp
index 27dcfdd8344..7fa7f1b27ec 100644
--- a/be/test/exprs/expr_zonemap_filter_test.cpp
+++ b/be/test/exprs/expr_zonemap_filter_test.cpp
@@ -33,14 +33,18 @@
#include "common/object_pool.h"
#include "core/column/column_vector.h"
#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_bitmap.h"
#include "core/data_type/data_type_date_or_datetime_v2.h"
#include "core/data_type/data_type_decimal.h"
+#include "core/data_type/data_type_factory.hpp"
+#include "core/data_type/data_type_jsonb.h"
#include "core/data_type/data_type_nullable.h"
#include "core/data_type/data_type_number.h"
#include "core/data_type/data_type_string.h"
#include "core/data_type/data_type_struct.h"
#include "core/data_type/data_type_time.h"
#include "core/data_type/data_type_varbinary.h"
+#include "core/data_type/data_type_variant.h"
#include "core/field.h"
#include "core/string_ref.h"
#include "core/value/vdatetime_value.h"
@@ -142,6 +146,23 @@ ZoneMapEvalContext make_context(segment_v2::ZoneMap
zone_map, const DataTypePtr&
return ctx;
}
+// Two slots in one context, which is what the segment-level and Parquet
row-group-level callers
+// build for a compound expression. Slot 0 is the left operand, slot 1 the
right.
+ZoneMapEvalContext make_two_slot_context(segment_v2::ZoneMap left,
segment_v2::ZoneMap right,
+ const DataTypePtr& left_type,
+ const DataTypePtr& right_type) {
+ ZoneMapEvalContext ctx;
+ ZoneMapEvalContext::SlotZoneMap left_slot;
+ left_slot.data_type = left_type;
+ left_slot.zone_map =
std::make_shared<segment_v2::ZoneMap>(std::move(left));
+ ctx.slots.emplace(0, std::move(left_slot));
+ ZoneMapEvalContext::SlotZoneMap right_slot;
+ right_slot.data_type = right_type;
+ right_slot.zone_map =
std::make_shared<segment_v2::ZoneMap>(std::move(right));
+ ctx.slots.emplace(1, std::move(right_slot));
+ return ctx;
+}
+
DictionaryEvalContext make_dictionary_context(std::vector<Field> values,
const DataTypePtr& data_type) {
DictionaryEvalContext ctx;
@@ -365,6 +386,28 @@ private:
std::string _expr_name = "unsupported_single_slot_expr";
};
+// A stand-in for a binary comparison predicate node (`a < b`, `a = 5`, ...).
+// contains_slot_slot_comparison inspects only children(), so the concrete
node type does not matter:
+// a comparison's two operands are its children.
+class BinaryLeafExpr final : public VExpr {
+public:
+ BinaryLeafExpr(VExprSPtr left, VExprSPtr right)
+ : VExpr(std::make_shared<DataTypeUInt8>(), false) {
+ add_child(std::move(left));
+ add_child(std::move(right));
+ }
+
+ const std::string& expr_name() const override { return _expr_name; }
+
+ Status execute_column_impl(VExprContext*, const Block*, const Selector*,
size_t,
+ ColumnPtr&) const override {
+ return Status::InternalError("BinaryLeafExpr is only used by zonemap
tests");
+ }
+
+private:
+ std::string _expr_name = "binary_leaf_expr";
+};
+
VExprSPtr make_fixed_zonemap_expr(ZoneMapFilterResult result) {
return std::make_shared<FixedZonemapExpr>(result);
}
@@ -1969,4 +2012,426 @@ TEST(ExprZonemapFilterTest,
ExprContextZonemapEvaluationShortCircuitsOnNoMatch)
EXPECT_EQ(0, ctx.stats.unusable_zonemap_eval_count);
}
+TEST(ExprZonemapFilterTest,
SlotSlotNotEqualsPrunesOnlyWhenBothCollapseToOneValue) {
+ auto type = int_type();
+ auto left = make_slot(0, type);
+ auto right = make_slot(1, type);
+ FunctionComparison<NotEqualsOp, NameNotEquals> not_equals;
+
+ // Both columns hold the single value 5, so `a != b` is false on every row.
+ auto same_point =
+ make_two_slot_context(make_int_zonemap(5, 5), make_int_zonemap(5,
5), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ not_equals.evaluate_zonemap_filter(same_point, {left, right}));
+
+ // Two different single values: `a != b` is true everywhere, so nothing
can be pruned.
+ auto other_point =
+ make_two_slot_context(make_int_zonemap(5, 5), make_int_zonemap(6,
6), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ not_equals.evaluate_zonemap_filter(other_point, {left, right}));
+
+ // A non-degenerate range on either side leaves the outcome unknown.
+ auto left_range =
+ make_two_slot_context(make_int_zonemap(5, 6), make_int_zonemap(5,
5), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ not_equals.evaluate_zonemap_filter(left_range, {left, right}));
+ auto right_range =
+ make_two_slot_context(make_int_zonemap(5, 5), make_int_zonemap(5,
6), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ not_equals.evaluate_zonemap_filter(right_range, {left, right}));
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotCoversAllSixOperatorsAtTheBoundary) {
+ auto type = int_type();
+ auto left = make_slot(0, type);
+ auto right = make_slot(1, type);
+ // Left entirely above right, ranges touching at 10: [10, 20] vs [1, 10].
+ auto ctx = make_two_slot_context(make_int_zonemap(10, 20),
make_int_zonemap(1, 10), type, type);
+
+ // a < b needs min_a >= max_b, which holds at the touching bound.
+ FunctionComparison<LessOp, NameLess> less;
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch, less.evaluate_zonemap_filter(ctx,
{left, right}));
+ // a <= b needs min_a > max_b, which does not hold: a == b == 10 is
possible.
+ FunctionComparison<LessOrEqualsOp, NameLessOrEquals> less_equals;
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ less_equals.evaluate_zonemap_filter(ctx, {left, right}));
+ FunctionComparison<GreaterOp, NameGreater> greater;
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
greater.evaluate_zonemap_filter(ctx, {left, right}));
+ FunctionComparison<EqualsOp, NameEquals> equals;
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
equals.evaluate_zonemap_filter(ctx, {left, right}));
+
+ // Fully disjoint: equality becomes impossible and so does a <= b.
+ auto disjoint =
+ make_two_slot_context(make_int_zonemap(10, 20),
make_int_zonemap(1, 9), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ equals.evaluate_zonemap_filter(disjoint, {left, right}));
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ less_equals.evaluate_zonemap_filter(disjoint, {left, right}));
+
+ // Mirrored, so a > b and a >= b become impossible.
+ auto mirrored =
+ make_two_slot_context(make_int_zonemap(1, 9), make_int_zonemap(10,
20), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ greater.evaluate_zonemap_filter(mirrored, {left, right}));
+ FunctionComparison<GreaterOrEqualsOp, NameGreaterOrEquals> greater_equals;
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ greater_equals.evaluate_zonemap_filter(mirrored, {left, right}));
+
+ // Partially overlapping ranges: every operator must keep the zone. These
are the cases that
+ // pin down which end of each range the rule reads. Taking max_a instead
of min_a for a < b, or
+ // max_b instead of min_b for a > b, still prunes here and would drop the
overlapping rows.
+ // [10, 20] vs [5, 15] overlap on [10, 15]: a = 12, b = 14 satisfies a < b
and a <= b.
+ auto left_high =
+ make_two_slot_context(make_int_zonemap(10, 20),
make_int_zonemap(5, 15), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ less.evaluate_zonemap_filter(left_high, {left, right}));
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ less_equals.evaluate_zonemap_filter(left_high, {left, right}));
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ equals.evaluate_zonemap_filter(left_high, {left, right}));
+ // [5, 15] vs [10, 20]: a = 14, b = 12 satisfies a > b and a >= b.
+ auto right_high =
+ make_two_slot_context(make_int_zonemap(5, 15),
make_int_zonemap(10, 20), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ greater.evaluate_zonemap_filter(right_high, {left, right}));
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ greater_equals.evaluate_zonemap_filter(right_high, {left,
right}));
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ equals.evaluate_zonemap_filter(right_high, {left, right}));
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotGuardsCoverNullAndMissingStatistics) {
+ auto type = int_type();
+ auto left = make_slot(0, type);
+ auto right = make_slot(1, type);
+ FunctionComparison<LessOp, NameLess> less;
+
+ // A column with no non-null value makes the comparison NULL on every row.
Production builds
+ // such a zone map with default-constructed min/max, which carry TYPE_NULL
rather than the
+ // column type. That is why the has_not_null check has to run before the
range check: the range
+ // check asserts the bounds match the slot type and would fatal on
TYPE_NULL.
+ segment_v2::ZoneMap all_null_zonemap;
+ all_null_zonemap.has_null = true;
+ all_null_zonemap.has_not_null = false;
+ auto left_all_null =
+ make_two_slot_context(all_null_zonemap, make_int_zonemap(1, 9),
type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ less.evaluate_zonemap_filter(left_all_null, {left, right}));
+ EXPECT_EQ(0, left_all_null.stats.unusable_zonemap_eval_count);
+ // The same on the right operand: both halves of the check are
load-bearing.
+ auto right_all_null =
+ make_two_slot_context(make_int_zonemap(1, 9), all_null_zonemap,
type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ less.evaluate_zonemap_filter(right_all_null, {left, right}));
+ EXPECT_EQ(0, right_all_null.stats.unusable_zonemap_eval_count);
+
+ // The right slot is absent from the context entirely.
+ auto missing_slot = make_context(make_int_zonemap(1, 9), type);
+ EXPECT_EQ(ZoneMapFilterResult::kUnsupported,
+ less.evaluate_zonemap_filter(missing_slot, {left, right}));
+ EXPECT_EQ(1, missing_slot.stats.unusable_zonemap_eval_count);
+
+ // The right slot is present with a type but without a zone map.
+ auto missing_zonemap = make_context(make_int_zonemap(1, 9), type);
+ ZoneMapEvalContext::SlotZoneMap slot_without_zonemap;
+ slot_without_zonemap.data_type = type;
+ missing_zonemap.slots.emplace(1, std::move(slot_without_zonemap));
+ EXPECT_EQ(ZoneMapFilterResult::kUnsupported,
+ less.evaluate_zonemap_filter(missing_zonemap, {left, right}));
+ EXPECT_EQ(1, missing_zonemap.stats.unusable_zonemap_eval_count);
+
+ // pass_all marks the bounds as unusable even though has_not_null is set.
+ auto pass_all_zonemap = make_int_zonemap(1, 9);
+ pass_all_zonemap.pass_all = true;
+ auto pass_all = make_two_slot_context(make_int_zonemap(10, 20),
std::move(pass_all_zonemap),
+ type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kUnsupported,
+ less.evaluate_zonemap_filter(pass_all, {left, right}));
+ EXPECT_EQ(1, pass_all.stats.unusable_zonemap_eval_count);
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotHandlesTheSameSlotOnBothSides) {
+ // FE's SimplifySelfComparison rewrites `a != a` (parsed as Not(EqualTo(a,
a))) by folding the
+ // inner equality, so it does not normally reach the BE as a slot-vs-slot
NotEqualTo. This pins
+ // the conservative fallback if such a shape still reaches the evaluator:
the generic point-range
+ // rule proves kNoMatch when a is a single value, and a range stays
kMayMatch instead of being
+ // special-cased as self-comparison.
+ auto type = int_type();
+ auto slot = make_slot(0, type);
+ FunctionComparison<NotEqualsOp, NameNotEquals> not_equals;
+
+ auto single_value = make_context(make_int_zonemap(7, 7), type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ not_equals.evaluate_zonemap_filter(single_value, {slot, slot}));
+ auto range = make_context(make_int_zonemap(7, 8), type);
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ not_equals.evaluate_zonemap_filter(range, {slot, slot}));
+}
+
+TEST(ExprZonemapFilterTest, ContainsSlotSlotComparisonFindsSlotPairsAnywhere) {
+ auto type = int_type();
+ auto slot_a = make_slot(0, type);
+ auto slot_b = make_slot(1, type);
+
+ EXPECT_FALSE(expr_zonemap::contains_slot_slot_comparison(nullptr));
+
+ // `a < b`: two distinct slots at the root.
+ EXPECT_TRUE(expr_zonemap::contains_slot_slot_comparison(
+ std::make_shared<BinaryLeafExpr>(slot_a, slot_b)));
+
+ // `a < a`: same column on both sides. This is the case a distinct-column
count misses -- one
+ // column id, still a slot-vs-slot leaf -- and the regression this gate
has to catch.
+ EXPECT_TRUE(expr_zonemap::contains_slot_slot_comparison(
+ std::make_shared<BinaryLeafExpr>(slot_a, slot_a)));
+
+ // `a < 5`: slot versus literal is not a slot-vs-slot comparison.
+ EXPECT_FALSE(expr_zonemap::contains_slot_slot_comparison(
+ std::make_shared<BinaryLeafExpr>(slot_a, make_int_literal(5))));
+
+ // `a < 5 AND c < d`: the slot pair is nested on the second child of the
AND.
+ auto slot_c = make_slot(2, type);
+ auto slot_d = make_slot(3, type);
+ auto and_pred =
+
std::make_shared<VCompoundPred>(make_compound_node(TExprOpcode::COMPOUND_AND,
2));
+ and_pred->add_child(std::make_shared<BinaryLeafExpr>(slot_a,
make_int_literal(5)));
+ and_pred->add_child(std::make_shared<BinaryLeafExpr>(slot_c, slot_d));
+ EXPECT_TRUE(expr_zonemap::contains_slot_slot_comparison(and_pred));
+
+ // `a < 5 OR b < 6`: only slot-vs-literal leaves, so no slot pair anywhere
in the tree.
+ auto or_pred =
std::make_shared<VCompoundPred>(make_compound_node(TExprOpcode::COMPOUND_OR,
2));
+ or_pred->add_child(std::make_shared<BinaryLeafExpr>(slot_a,
make_int_literal(5)));
+ or_pred->add_child(std::make_shared<BinaryLeafExpr>(slot_b,
make_int_literal(6)));
+ EXPECT_FALSE(expr_zonemap::contains_slot_slot_comparison(or_pred));
+}
+
+TEST(ExprZonemapFilterTest,
SlotSlotWidensOnlyTheZonemapGateNotDictionaryOrBloom) {
+ // This is the regression guard for the trap in this change:
comparison_zonemap_detail's
+ // can_evaluate is shared by dictionary filtering and
can_evaluate_equality, and both dereference
+ // extract_slot_and_literal behind a DORIS_CHECK. Widening it would abort
on `a != b`, so the
+ // slot-vs-slot shape must only ever be accepted by
can_evaluate_zonemap_filter.
+ auto type = int_type();
+ auto left = make_slot(0, type);
+ auto right = make_slot(1, type);
+ FunctionComparison<NotEqualsOp, NameNotEquals> not_equals;
+
+ EXPECT_TRUE(not_equals.can_evaluate_zonemap_filter({left, right}));
+ EXPECT_FALSE(not_equals.can_evaluate_dictionary_filter({left, right}));
+
+ // The bloom gate has to be checked through EQ. can_evaluate_bloom_filter
starts with
+ // `op == Op::EQ`, so asserting it on NE would pass no matter what the
rest of the gate does.
+ FunctionComparison<EqualsOp, NameEquals> equals;
+ EXPECT_TRUE(equals.can_evaluate_zonemap_filter({left, right}));
+ EXPECT_FALSE(equals.can_evaluate_bloom_filter({left, right}));
+ EXPECT_TRUE(equals.can_evaluate_bloom_filter({left, make_int_literal(1)}));
+
+ // Slot vs literal keeps working on every gate.
+ EXPECT_TRUE(not_equals.can_evaluate_zonemap_filter({left,
make_int_literal(1)}));
+ EXPECT_TRUE(not_equals.can_evaluate_dictionary_filter({left,
make_int_literal(1)}));
+
+ // Two columns of incompatible types are rejected: the two Fields would be
compared directly and
+ // Field comparison throws on mismatched non-string types.
+ auto bigint = std::make_shared<DataTypeInt64>();
+ EXPECT_FALSE(not_equals.can_evaluate_zonemap_filter({left, make_slot(1,
bigint)}));
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotGateRejectsVarbinaryButAllowsStrings) {
+ // VARBINARY is rejected at the gate: the v1 Parquet path produces a
TYPE_STRING bound for a
+ // VARBINARY slot, so evaluating it would abort in
fetch_compatible_slot_type instead of falling
+ // back. String and int pairs are admitted; the string bound-length safety
is in evaluation.
+ auto string_type = std::make_shared<DataTypeString>();
+ auto varbinary_type = std::make_shared<DataTypeVarbinary>();
+ FunctionComparison<GreaterOp, NameGreater> greater;
+
+ EXPECT_FALSE(greater.can_evaluate_zonemap_filter(
+ {make_slot(0, varbinary_type), make_slot(1, varbinary_type)}));
+
+ EXPECT_TRUE(greater.can_evaluate_zonemap_filter(
+ {make_slot(0, string_type), make_slot(1, string_type)}));
+ auto type = int_type();
+ EXPECT_TRUE(greater.can_evaluate_zonemap_filter({make_slot(0, type),
make_slot(1, type)}));
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotGateRejectsVariantAndUnorderedTypes) {
+ // A segment types a VARIANT path slot by its inferred leaf type, so
admitting a VARIANT pair
+ // would fail fetch_compatible_slot_type's check. Complex, JSONB and
aggregate-state Fields all
+ // compare equal, so their bounds prove nothing. Each is rejected,
nullable or not.
+ FunctionComparison<EqualsOp, NameEquals> equals;
+ const std::vector<DataTypePtr> excluded_types = {
+ std::make_shared<DataTypeVariant>(),
+ std::make_shared<DataTypeJsonb>(),
+ std::make_shared<DataTypeArray>(make_nullable(int_type())),
+ std::make_shared<DataTypeBitMap>(),
+ };
+ for (const auto& excluded : excluded_types) {
+ for (const auto& type : {excluded, make_nullable(excluded)}) {
+ EXPECT_FALSE(
+ equals.can_evaluate_zonemap_filter({make_slot(0, type),
make_slot(1, type)}))
+ << type->get_name();
+ }
+ }
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotStringPrunesUntruncatedBoundsOnly) {
+ // A native string zone-map max of exactly 512 bytes was bumped upward and
may also have been
+ // truncated, so it may not be a real upper bound. A shorter bound is the
exact value.
+ // left in ["a","c"], right pinned above "c", so `left > right` cannot
hold when the bounds
+ // are trusted.
+ auto string_type = std::make_shared<DataTypeString>();
+ auto left = make_slot(0, string_type);
+ auto right = make_slot(1, string_type);
+ FunctionComparison<GreaterOp, NameGreater> greater;
+
+ auto eval = [&](const std::string& r) {
+ auto ctx = make_two_slot_context(make_string_zonemap("a", "c"),
make_string_zonemap(r, r),
+ string_type, string_type);
+ return greater.evaluate_zonemap_filter(ctx, {left, right});
+ };
+
+ // Short and 511-byte bounds are untruncated, so pruning stands.
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch, eval("m"));
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch, eval(std::string(511, 'x')));
+ // A 512-byte bound may be truncated/bumped, and a longer one is out of
spec: fall back.
+ EXPECT_EQ(ZoneMapFilterResult::kUnsupported, eval(std::string(512, 'x')));
+ EXPECT_EQ(ZoneMapFilterResult::kUnsupported, eval(std::string(513, 'x')));
+ // Before the carry fix the writer bumped only the last byte, so a 0xff
there wrapped to 0x00
+ // and left a 512-byte bound ending in '\0' below the data; still length
512, still unusable.
+ std::string wrapped(511, 'x');
+ wrapped.push_back('\0');
+ EXPECT_EQ(ZoneMapFilterResult::kUnsupported, eval(wrapped));
+
+ // An embedded NUL counts toward .size(); a short bound with one stays
usable.
+ std::string with_nul("a\0b", 3);
+ auto ctx_nul = make_two_slot_context(make_string_zonemap(with_nul,
with_nul),
+ make_string_zonemap("m", "z"),
string_type, string_type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ greater.evaluate_zonemap_filter(ctx_nul, {left, right}));
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotBailsOutWhenAFloatingNanCountIsUnknown) {
+ // Parquet bounds omit NaN without reporting how many were skipped, so a
zone map that looks like
+ // a single point may still hide one. That would flip the NE rule from
false to true, so the
+ // slot-vs-slot path refuses to prune for every operator once either side
is marked unknown.
+ auto type = std::make_shared<DataTypeFloat64>();
+ auto left = make_slot(0, type);
+ auto right = make_slot(1, type);
+
+ auto make_double_zonemap = [](double value) {
+ segment_v2::ZoneMap zone_map;
+ zone_map.min_value = Field::create_field<TYPE_DOUBLE>(value);
+ zone_map.max_value = Field::create_field<TYPE_DOUBLE>(value);
+ zone_map.has_not_null = true;
+ return zone_map;
+ };
+
+ // Control group. Both sides are the same single value, so NE prunes
without the guard; the
+ // contrast with the loop below is what shows the guard is what changes
the outcome. The counter
+ // alone cannot show that, since every bail-out path in evaluate_slot_slot
bumps the same one.
+ FunctionComparison<NotEqualsOp, NameNotEquals> not_equals;
+ auto known =
+ make_two_slot_context(make_double_zonemap(1.0),
make_double_zonemap(1.0), type, type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ not_equals.evaluate_zonemap_filter(known, {left, right}));
+ EXPECT_EQ(0, known.stats.unusable_zonemap_eval_count);
+
+ auto expect_bails_out = [&](auto&& comparison, const char* predicate) {
+ for (int unknown_slot : {0, 1}) {
+ auto ctx = make_two_slot_context(make_double_zonemap(1.0),
make_double_zonemap(1.0),
+ type, type);
+ ctx.slots[unknown_slot].floating_nan_count_unknown = true;
+ EXPECT_EQ(ZoneMapFilterResult::kUnsupported,
+ comparison.evaluate_zonemap_filter(ctx, {left, right}))
+ << predicate << " must stop pruning when slot " <<
unknown_slot
+ << " has an unknown NaN count";
+ EXPECT_EQ(1, ctx.stats.unusable_zonemap_eval_count);
+ }
+ };
+
+ // All six operators bail out. On these bounds NE, LT and GT would
otherwise prune and EQ, LE
+ // and GE would otherwise keep the zone, so none of the six is vacuously
covered.
+ expect_bails_out(FunctionComparison<EqualsOp, NameEquals> {}, "a = b");
+ expect_bails_out(FunctionComparison<NotEqualsOp, NameNotEquals> {}, "a !=
b");
+ expect_bails_out(FunctionComparison<LessOp, NameLess> {}, "a < b");
+ expect_bails_out(FunctionComparison<LessOrEqualsOp, NameLessOrEquals> {},
"a <= b");
+ expect_bails_out(FunctionComparison<GreaterOp, NameGreater> {}, "a > b");
+ expect_bails_out(FunctionComparison<GreaterOrEqualsOp,
NameGreaterOrEquals> {}, "a >= b");
+}
+
+TEST(ExprZonemapFilterTest, SlotSlotComparesDecimalBoundsInOneDomain) {
+ // Decimal bounds are unscaled payloads, so the rule is only meaningful
when both sides share a
+ // scale. The v1 Parquet reader decodes row-group bounds in the file's own
logical type while
+ // _type_matches compares only primitive types, so it now leaves the zone
map out when the file
+ // and table types differ; this pins the behaviour the rule needs when
they agree.
+ auto decimal_type =
DataTypeFactory::instance().create_data_type(TYPE_DECIMAL64, false, 18, 2);
+ auto left = make_slot(0, decimal_type);
+ auto right = make_slot(1, decimal_type);
+ FunctionComparison<LessOp, NameLess> less;
+
+ auto make_decimal_zonemap = [](int64_t unscaled_min, int64_t unscaled_max)
{
+ segment_v2::ZoneMap zone_map;
+ zone_map.min_value =
Field::create_field<TYPE_DECIMAL64>(Decimal64(unscaled_min));
+ zone_map.max_value =
Field::create_field<TYPE_DECIMAL64>(Decimal64(unscaled_max));
+ zone_map.has_not_null = true;
+ return zone_map;
+ };
+
+ // 10.00 .. 20.00 against 0.50 .. 1.00: a < b cannot hold.
+ auto separated =
+ make_two_slot_context(make_decimal_zonemap(1000, 2000),
make_decimal_zonemap(50, 100),
+ decimal_type, decimal_type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
+ less.evaluate_zonemap_filter(separated, {left, right}));
+
+ // Touching at 10.00, which is what separates `lmin >= rmax` from `lmin >
rmax`: a < b still
+ // cannot hold, so LT must prune while LE must not.
+ auto touching =
+ make_two_slot_context(make_decimal_zonemap(1000, 2000),
make_decimal_zonemap(500, 1000),
+ decimal_type, decimal_type);
+ EXPECT_EQ(ZoneMapFilterResult::kNoMatch,
less.evaluate_zonemap_filter(touching, {left, right}));
+ FunctionComparison<LessOrEqualsOp, NameLessOrEquals> less_equals;
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ less_equals.evaluate_zonemap_filter(touching, {left, right}));
+
+ // Overlapping in the same scale: nothing can be concluded.
+ auto overlapping =
+ make_two_slot_context(make_decimal_zonemap(1000, 2000),
+ make_decimal_zonemap(1500, 2500),
decimal_type, decimal_type);
+ EXPECT_EQ(ZoneMapFilterResult::kMayMatch,
+ less.evaluate_zonemap_filter(overlapping, {left, right}));
+}
+
+TEST(ExprZonemapFilterTest, ParquetSlotZoneMapMarksFloatingNanCountUnknown) {
+ // The rule used to be spelled out in format_v2's add_slot_zonemap only,
and the v1 Parquet
+ // reader assigned data_type directly and so left the flag false. Two
double columns whose
+ // Parquet bounds both read [1.0, 1.0] then got pruned by `a != b` even
when one of them hid a
+ // NaN. Both readers now go through this setter, so this test pins the
rule down in one place.
+ ZoneMapEvalContext::SlotZoneMap double_slot;
+
double_slot.set_data_type_from_parquet(std::make_shared<DataTypeFloat64>());
+ EXPECT_TRUE(double_slot.floating_nan_count_unknown);
+
+ ZoneMapEvalContext::SlotZoneMap float_slot;
+ float_slot.set_data_type_from_parquet(std::make_shared<DataTypeFloat32>());
+ EXPECT_TRUE(float_slot.floating_nan_count_unknown);
+
+ // Nullable wrappers must not hide the floating type.
+ ZoneMapEvalContext::SlotZoneMap nullable_double_slot;
+ nullable_double_slot.set_data_type_from_parquet(
+
std::make_shared<DataTypeNullable>(std::make_shared<DataTypeFloat64>()));
+ EXPECT_TRUE(nullable_double_slot.floating_nan_count_unknown);
+
+ // Non-floating columns keep exact bounds, so they must stay prunable.
+ ZoneMapEvalContext::SlotZoneMap int_slot;
+ auto int_data_type = int_type();
+ int_slot.set_data_type_from_parquet(int_data_type);
+ EXPECT_FALSE(int_slot.floating_nan_count_unknown);
+ EXPECT_EQ(int_data_type, int_slot.data_type);
+
+ // A slot with no type at all is copied through as before rather than
dereferenced. The type is
+ // null only for SlotDescriptor's test-only default constructor, and the
evaluator treats a slot
+ // without a type as unusable anyway.
+ ZoneMapEvalContext::SlotZoneMap untyped_slot;
+ untyped_slot.set_data_type_from_parquet(nullptr);
+ EXPECT_EQ(nullptr, untyped_slot.data_type);
+ EXPECT_FALSE(untyped_slot.floating_nan_count_unknown);
+}
+
} // namespace doris
diff --git
a/regression-test/data/query_p0/expr_zonemap/test_expr_zonemap_pruning.out
b/regression-test/data/query_p0/expr_zonemap/test_expr_zonemap_pruning.out
new file mode 100644
index 00000000000..dc035390d2b
--- /dev/null
+++ b/regression-test/data/query_p0/expr_zonemap/test_expr_zonemap_pruning.out
@@ -0,0 +1,132 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !matched_rows --
+4096
+
+-- !starts_with_pruned --
+0
+
+-- !in_pruned --
+0
+
+-- !is_null_pruned --
+0
+
+-- !comparison_pruned --
+0
+
+-- !page_reachability --
+4096
+
+-- !inferred_prefix_range --
+4096
+
+-- !inferred_date_range --
+4096
+
+-- !inferred_date_format --
+4096
+
+-- !inferred_date_format_equality --
+4096
+
+-- !inferred_direct_scan --
+4096
+
+-- !large_quarter_period --
+8192
+
+-- !char_pruned --
+0
+
+-- !is_not_null_pruned --
+0
+
+-- !two_column_gt --
+0
+
+-- !two_column_ge --
+0
+
+-- !two_column_lt --
+0
+
+-- !two_column_le --
+0
+
+-- !two_column_eq --
+0
+
+-- !two_column_ne --
+0
+
+-- !overlap_lo_lt_alt --
+2048
+
+-- !overlap_lo_ne_alt --
+2048
+
+-- !overlap_lo_eq_alt --
+2048
+
+-- !overlap_lo_le_alt --
+4096
+
+-- !overlap_lo_gt_alt --
+0
+
+-- !overlap_lo_ge_alt_shift --
+4096
+
+-- !overlap_lo_lt_cast_hi --
+4096
+
+-- !two_column_null --
+0
+
+-- !two_column_all_null --
+0
+
+-- !per_segment_ne --
+1024
+
+-- !per_segment_eq --
+2048
+
+-- !per_segment_ne_without --
+1024
+
+-- !per_segment_eq_without --
+2048
+
+-- !literal_still_prunes --
+0
+
+-- !column_predicate --
+0
+
+-- !double_pruned --
+0
+
+-- !nan_lo_gt_hi --
+3
+
+-- !nan_lo_ne_hi --
+6
+
+-- !nan_lo_eq_hi --
+0
+
+-- !nan_lo_lt_hi --
+3
+
+-- !variant_path_eq --
+1
+3
+
+-- !variant_path_ne --
+2
+4
+
+-- !hidden_version_or_miss --
+2048
+
diff --git
a/regression-test/suites/query_p0/expr_zonemap/test_expr_zonemap_pruning.groovy
b/regression-test/suites/query_p0/expr_zonemap/test_expr_zonemap_pruning.groovy
index abad8c97d92..00d5012efa9 100644
---
a/regression-test/suites/query_p0/expr_zonemap/test_expr_zonemap_pruning.groovy
+++
b/regression-test/suites/query_p0/expr_zonemap/test_expr_zonemap_pruning.groovy
@@ -48,16 +48,11 @@ suite("test_expr_zonemap_pruning") {
def profileAction = new ProfileAction(context)
def getProfileByToken = { String token ->
- for (int retry = 0; retry < 10; ++retry) {
- List profileData = profileAction.getProfileList()
- for (final def profileItem in profileData) {
- if (profileItem["Sql Statement"].toString().contains(token)) {
- return profileAction.getProfile(profileItem["Profile
ID"].toString())
- }
- }
- Thread.sleep(500)
- }
- throw new IllegalStateException("Missing profile for token: " + token)
+ // Wait for the profile to reach COMPLETE before reading counters. A
profile can be listed and
+ // fetchable while still aggregating across instances, so an exact
counter (the 2 / 1 / 3
+ // filtered-segment assertions below) could otherwise read a partial
total. getProfileBySql
+ // polls Profile Completion State and returns only once it is COMPLETE.
+ return profileAction.getProfileBySql(token)
}
def counterSum = { String profile, String counterName ->
@@ -104,40 +99,50 @@ suite("test_expr_zonemap_pruning") {
assertTrue(counterValue > 0)
}
- def matchedRows = sql """
+ qt_matched_rows """
SELECT COUNT(*) FROM test_expr_zonemap_pruning WHERE starts_with(v,
'a')
"""
- assertEquals(4096L, matchedRows[0][0] as long)
+ qt_starts_with_pruned """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning WHERE starts_with(v,
'z')
+ """
def startsWithToken = "expr_zonemap_pruning_starts_with_" +
UUID.randomUUID().toString()
- def startsWithPrunedRows = sql """
+ sql """
SELECT '${startsWithToken}', COUNT(*) FROM test_expr_zonemap_pruning
WHERE starts_with(v, 'z')
"""
- assertEquals(0L, startsWithPrunedRows[0][1] as long)
assertExprZonemapPruned(startsWithToken)
+ qt_in_pruned """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning
+ WHERE v IN ('z0', 'z1') OR starts_with(v, 'zz')
+ """
def inToken = "expr_zonemap_pruning_in_" + UUID.randomUUID().toString()
- def inPrunedRows = sql """
+ sql """
SELECT '${inToken}', COUNT(*) FROM test_expr_zonemap_pruning
WHERE v IN ('z0', 'z1') OR starts_with(v, 'zz')
"""
- assertEquals(0L, inPrunedRows[0][1] as long)
assertExprZonemapPruned(inToken)
+ qt_is_null_pruned """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning
+ WHERE v IS NULL OR starts_with(v, 'zz')
+ """
def nullToken = "expr_zonemap_pruning_is_null_" +
UUID.randomUUID().toString()
- def nullPrunedRows = sql """
+ sql """
SELECT '${nullToken}', COUNT(*) FROM test_expr_zonemap_pruning
WHERE v IS NULL OR starts_with(v, 'zz')
"""
- assertEquals(0L, nullPrunedRows[0][1] as long)
assertExprZonemapPruned(nullToken)
+ qt_comparison_pruned """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning
+ WHERE id > 5000 OR id < 0
+ """
def comparisonToken = "expr_zonemap_pruning_comparison_" +
UUID.randomUUID().toString()
- def comparisonPrunedRows = sql """
+ sql """
SELECT '${comparisonToken}', COUNT(*) FROM test_expr_zonemap_pruning
WHERE id > 5000 OR id < 0
"""
- assertEquals(0L, comparisonPrunedRows[0][1] as long)
assertExprZonemapPruned(comparisonToken)
sql """ DROP TABLE IF EXISTS test_expr_zonemap_page_reachability """
@@ -165,12 +170,15 @@ suite("test_expr_zonemap_pruning") {
"""
sql """ sync """
+ qt_page_reachability """
+ SELECT COUNT(*) FROM test_expr_zonemap_page_reachability
+ WHERE starts_with(v, 'm_')
+ """
def pageReachabilityToken = "expr_zonemap_pruning_page_reachability_" +
UUID.randomUUID().toString()
- def pageReachabilityRows = sql """
+ sql """
SELECT '${pageReachabilityToken}', COUNT(*) FROM
test_expr_zonemap_page_reachability
WHERE starts_with(v, 'm_')
"""
- assertEquals(4096L, pageReachabilityRows[0][1] as long)
assertProfileCounterPositive(pageReachabilityToken,
"ExprZoneMapFilteredPages")
sql """ DROP TABLE IF EXISTS test_monotonic_zonemap """
@@ -220,37 +228,49 @@ suite("test_expr_zonemap_pruning") {
contains "pushAggOp=COUNT_ON_INDEX"
}
+ qt_inferred_prefix_range """
+ SELECT COUNT(*) FROM test_monotonic_zonemap
+ WHERE substring(v, 1, 1) = 'a'
+ """
def inferredPrefixRangeToken =
"expr_zonemap_pruning_inferred_prefix_range_" + UUID.randomUUID().toString()
- def inferredPrefixRangeRows = sql """
+ sql """
SELECT '${inferredPrefixRangeToken}', COUNT(*) FROM
test_monotonic_zonemap
WHERE substring(v, 1, 1) = 'a'
"""
- assertEquals(4096L, inferredPrefixRangeRows[0][1] as long)
assertProfileCounterPositive(inferredPrefixRangeToken, "RowsStatsFiltered")
+ qt_inferred_date_range """
+ SELECT COUNT(*) FROM test_monotonic_zonemap
+ WHERE date_trunc(dt, 'day') = '2026-07-28 00:00:00'
+ """
def inferredDateRangeToken = "expr_zonemap_pruning_inferred_date_range_" +
UUID.randomUUID().toString()
- def inferredDateRangeRows = sql """
+ sql """
SELECT '${inferredDateRangeToken}', COUNT(*) FROM
test_monotonic_zonemap
WHERE date_trunc(dt, 'day') = '2026-07-28 00:00:00'
"""
- assertEquals(4096L, inferredDateRangeRows[0][1] as long)
assertProfileCounterPositive(inferredDateRangeToken, "RowsStatsFiltered")
+ qt_inferred_date_format """
+ SELECT COUNT(*) FROM test_monotonic_zonemap
+ WHERE date_format(dt, '%Y-%m-%d') >= '2026-08-01'
+ """
def inferredDateFormatToken = "expr_zonemap_pruning_inferred_date_format_"
+ UUID.randomUUID().toString()
- def inferredDateFormatRows = sql """
+ sql """
SELECT '${inferredDateFormatToken}', COUNT(*) FROM
test_monotonic_zonemap
WHERE date_format(dt, '%Y-%m-%d') >= '2026-08-01'
"""
- assertEquals(4096L, inferredDateFormatRows[0][1] as long)
assertProfileCounterPositive(inferredDateFormatToken, "RowsStatsFiltered")
+ qt_inferred_date_format_equality """
+ SELECT COUNT(*) FROM test_monotonic_zonemap
+ WHERE date_format(dt, '%Y-%m-%d') = '2026-07-28'
+ """
def inferredDateFormatEqualityToken =
"expr_zonemap_pruning_inferred_date_format_equality_" +
UUID.randomUUID().toString()
- def inferredDateFormatEqualityRows = sql """
+ sql """
SELECT '${inferredDateFormatEqualityToken}', COUNT(*) FROM
test_monotonic_zonemap
WHERE date_format(dt, '%Y-%m-%d') = '2026-07-28'
"""
- assertEquals(4096L, inferredDateFormatEqualityRows[0][1] as long)
assertProfileCounterPositive(inferredDateFormatEqualityToken,
"RowsStatsFiltered")
sql """ set enable_count_on_index_pushdown = false """
@@ -261,20 +281,22 @@ suite("test_expr_zonemap_pruning") {
"""
contains "pushAggOp=NONE"
}
+ qt_inferred_direct_scan """
+ SELECT COUNT(*) FROM test_monotonic_zonemap
+ WHERE date_trunc(dt, 'day') = '2026-07-28 00:00:00'
+ """
def inferredDirectScanToken =
"expr_zonemap_pruning_inferred_direct_scan_" +
UUID.randomUUID().toString()
- def inferredDirectScanRows = sql """
+ sql """
SELECT '${inferredDirectScanToken}', COUNT(*) FROM
test_monotonic_zonemap
WHERE date_trunc(dt, 'day') = '2026-07-28 00:00:00'
"""
- assertEquals(4096L, inferredDirectScanRows[0][1] as long)
assertProfileCounterPositive(inferredDirectScanToken, "RowsStatsFiltered")
- def largeQuarterPeriodRows = sql """
+ qt_large_quarter_period """
SELECT COUNT(*) FROM test_monotonic_zonemap
WHERE quarter_floor(dt, 1431655766) = '0001-01-01 00:00:00'
"""
- assertEquals(8192L, largeQuarterPeriodRows[0][0] as long)
sql """ set enable_count_on_index_pushdown = true """
sql """ set enable_expr_zonemap_filter = true """
@@ -298,12 +320,15 @@ suite("test_expr_zonemap_pruning") {
"""
sql """ sync """
+ qt_char_pruned """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_char
+ WHERE starts_with(c, 'z') OR c IN ('z0', 'z1') OR c = 'z2'
+ """
def charToken = "expr_zonemap_pruning_char_" + UUID.randomUUID().toString()
- def charPrunedRows = sql """
+ sql """
SELECT '${charToken}', COUNT(*) FROM test_expr_zonemap_pruning_char
WHERE starts_with(c, 'z') OR c IN ('z0', 'z1') OR c = 'z2'
"""
- assertEquals(0L, charPrunedRows[0][1] as long)
assertExprZonemapPruned(charToken)
sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_nulls """
@@ -322,11 +347,429 @@ suite("test_expr_zonemap_pruning") {
sql """ INSERT INTO test_expr_zonemap_pruning_nulls VALUES (1, NULL), (2,
NULL) """
sql """ sync """
+ qt_is_not_null_pruned """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_nulls
+ WHERE v IS NOT NULL OR starts_with(v, 'zz')
+ """
def isNotNullToken = "expr_zonemap_pruning_is_not_null_" +
UUID.randomUUID().toString()
- def isNotNullPrunedRows = sql """
+ sql """
SELECT '${isNotNullToken}', COUNT(*) FROM
test_expr_zonemap_pruning_nulls
WHERE v IS NOT NULL OR starts_with(v, 'zz')
"""
- assertEquals(0L, isNotNullPrunedRows[0][1] as long)
assertExprZonemapPruned(isNotNullToken)
+
+ // Column-vs-column comparisons. A predicate over two columns of the same
table never became a
+ // ColumnPredicate, so it reaches the scanner as a common expression and
is evaluated against the
+ // segment zone map of both slots at once.
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_two_columns """
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_two_columns (
+ id INT,
+ lo INT,
+ hi INT,
+ alt INT,
+ expected INT,
+ actual INT
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ // lo lands in [0, 4095] and hi in [10000, 14095], so the two ranges are
fully separated. alt
+ // lands in [0, 7095] and equals lo on even rows and lo + 3000 on odd
ones, so lo vs alt cannot
+ // be decided from the bounds and every row has to be evaluated. expected
and actual are both
+ // the constant 7.
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_two_columns
+ SELECT CAST(number AS INT),
+ CAST(number AS INT),
+ CAST(number + 10000 AS INT),
+ IF(number % 2 = 0, CAST(number AS INT), CAST(number + 3000 AS
INT)),
+ 7,
+ 7
+ FROM numbers("number" = "4096")
+ """
+ sql """ sync """
+
+ // Runs the same query with expr zonemap pruning on and off and asserts
the two agree. The
+ // counter only shows that pruning fired; this is what shows it fired
correctly.
+ def assertSameWithAndWithoutPruning = { String predicate ->
+ sql """ set enable_expr_zonemap_filter = false """
+ def withoutPruning = sql """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_two_columns WHERE
${predicate}
+ """
+ sql """ set enable_expr_zonemap_filter = true """
+ def withPruning = sql """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_two_columns WHERE
${predicate}
+ """
+ assertEquals(withoutPruning[0][0] as long, withPruning[0][0] as long)
+ return withPruning[0][0] as long
+ }
+
+ def assertTwoColumnPruned = { String predicate, String label ->
+ def token = "expr_zonemap_pruning_two_columns_" + label + "_" +
UUID.randomUUID().toString()
+ sql """
+ SELECT '${token}', COUNT(*) FROM
test_expr_zonemap_pruning_two_columns
+ WHERE ${predicate}
+ """
+ assertExprZonemapPruned(token)
+ assertSameWithAndWithoutPruning(predicate)
+ }
+
+ // lo > hi and lo >= hi: rejected because min(hi) is already above max(lo).
+ assertTwoColumnPruned("lo > hi", "gt")
+ qt_two_column_gt """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo > hi"""
+ assertTwoColumnPruned("lo >= hi", "ge")
+ qt_two_column_ge """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo >= hi"""
+ // hi < lo and hi <= lo: the mirrored rules.
+ assertTwoColumnPruned("hi < lo", "lt")
+ qt_two_column_lt """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE hi < lo"""
+ assertTwoColumnPruned("hi <= lo", "le")
+ qt_two_column_le """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE hi <= lo"""
+ // lo = hi: the ranges are disjoint, so no row can be equal.
+ assertTwoColumnPruned("lo = hi", "eq")
+ qt_two_column_eq """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo = hi"""
+ // expected != actual: both columns collapse to the single value 7, which
is the only shape that
+ // lets != prune.
+ assertTwoColumnPruned("expected != actual", "ne")
+ qt_two_column_ne """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE expected != actual"""
+
+ // Overlapping ranges must survive, and the row counts must be exact.
These are the cases that
+ // catch a rule reading the wrong end of a range: lo in [0, 4095] against
alt in [0, 7095] cannot
+ // be separated by the bounds, so all of these have to fall through to
per-row evaluation. The
+ // enabled-vs-disabled equality stays programmatic; the exact counts land
in golden output.
+ assertSameWithAndWithoutPruning("lo < alt")
+ qt_overlap_lo_lt_alt """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo < alt"""
+ assertSameWithAndWithoutPruning("lo != alt")
+ qt_overlap_lo_ne_alt """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo != alt"""
+ assertSameWithAndWithoutPruning("lo = alt")
+ qt_overlap_lo_eq_alt """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo = alt"""
+ assertSameWithAndWithoutPruning("lo <= alt")
+ qt_overlap_lo_le_alt """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo <= alt"""
+ assertSameWithAndWithoutPruning("lo > alt")
+ qt_overlap_lo_gt_alt """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo > alt"""
+ assertSameWithAndWithoutPruning("lo >= alt - 3000")
+ qt_overlap_lo_ge_alt_shift """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo >= alt - 3000"""
+ // A cast on either side is rejected by the capability gate, so this must
still return the right
+ // answer rather than being pruned on raw bounds.
+ assertSameWithAndWithoutPruning("lo < CAST(hi AS BIGINT)")
+ qt_overlap_lo_lt_cast_hi """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns WHERE lo < CAST(hi AS BIGINT)"""
+
+ // One side partially NULL. min/max summarize the non-null values only,
and a NULL row makes the
+ // comparison NULL, which never satisfies the conjunct, so the separated
ranges still prune.
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_two_columns_nulls
"""
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_two_columns_nulls (
+ id INT,
+ lo INT,
+ hi INT
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_two_columns_nulls
+ SELECT CAST(number AS INT),
+ IF(number % 8 = 0, NULL, CAST(number AS INT)),
+ CAST(number + 10000 AS INT)
+ FROM numbers("number" = "4096")
+ """
+ sql """ sync """
+
+ qt_two_column_null """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_two_columns_nulls
+ WHERE lo > hi
+ """
+ def twoColumnNullToken =
+ "expr_zonemap_pruning_two_columns_null_" +
UUID.randomUUID().toString()
+ sql """
+ SELECT '${twoColumnNullToken}', COUNT(*) FROM
test_expr_zonemap_pruning_two_columns_nulls
+ WHERE lo > hi
+ """
+ assertExprZonemapPruned(twoColumnNullToken)
+
+ // A column with no non-null value at all makes the comparison NULL on
every row.
+ sql """ DROP TABLE IF EXISTS
test_expr_zonemap_pruning_two_columns_all_null """
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_two_columns_all_null (
+ id INT,
+ lo INT,
+ hi INT
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_two_columns_all_null
+ SELECT CAST(number AS INT), NULL, CAST(number AS INT)
+ FROM numbers("number" = "4096")
+ """
+ sql """ sync """
+
+ qt_two_column_all_null """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_two_columns_all_null
+ WHERE lo < hi
+ """
+ def allNullToken =
+ "expr_zonemap_pruning_two_columns_all_null_" +
UUID.randomUUID().toString()
+ sql """
+ SELECT '${allNullToken}', COUNT(*) FROM
test_expr_zonemap_pruning_two_columns_all_null
+ WHERE lo < hi
+ """
+ assertExprZonemapPruned(allNullToken)
+
+ // Same as assertExprZonemapPruned but returns the count, so a case can
pin the exact number of
+ // segments that had to be dropped instead of only that pruning happened
at all.
+ def filteredSegmentsOf = { String token ->
+ long filteredSegments = 0
+ for (int retry = 0; retry < 20; ++retry) {
+ String profile = getProfileByToken(token).toString()
+ filteredSegments = counterSum(profile,
"ExprZoneMapFilteredSegments")
+ if (filteredSegments > 0) {
+ return filteredSegments
+ }
+ Thread.sleep(500)
+ }
+ return filteredSegments
+ }
+
+ // Three loads, one segment each, each segment holding a constant in both
columns. This is the
+ // shape where a cross-column rule has to decide per segment rather than
per table, so the exact
+ // number of dropped segments is the assertion that matters.
+ //
+ // batch | a | b | a != b | a = b
+ // 1 | a | a | both collapse | overlap, keep
+ // 2 | a | b | disjoint,keep | disjoint, drop
+ // 3 | b | b | both collapse | overlap, keep
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_per_segment """
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_per_segment (
+ id INT,
+ a VARCHAR(32),
+ b VARCHAR(32)
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_per_segment
+ SELECT CAST(number AS INT), 'a', 'a' FROM numbers("number" = "1024")
+ """
+ sql """ sync """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_per_segment
+ SELECT CAST(number AS INT), 'a', 'b' FROM numbers("number" = "1024")
+ """
+ sql """ sync """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_per_segment
+ SELECT CAST(number AS INT), 'b', 'b' FROM numbers("number" = "1024")
+ """
+ sql """ sync """
+
+ qt_per_segment_ne """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_per_segment WHERE a != b
+ """
+ def perSegmentNeToken = "expr_zonemap_pruning_per_segment_ne_" +
UUID.randomUUID().toString()
+ sql """
+ SELECT '${perSegmentNeToken}', COUNT(*) FROM
test_expr_zonemap_pruning_per_segment
+ WHERE a != b
+ """
+ assertEquals(2L, filteredSegmentsOf(perSegmentNeToken))
+
+ qt_per_segment_eq """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_per_segment WHERE a = b
+ """
+ def perSegmentEqToken = "expr_zonemap_pruning_per_segment_eq_" +
UUID.randomUUID().toString()
+ sql """
+ SELECT '${perSegmentEqToken}', COUNT(*) FROM
test_expr_zonemap_pruning_per_segment
+ WHERE a = b
+ """
+ assertEquals(1L, filteredSegmentsOf(perSegmentEqToken))
+
+ sql """ set enable_expr_zonemap_filter = false """
+ qt_per_segment_ne_without """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_per_segment WHERE a != b
+ """
+ qt_per_segment_eq_without """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_per_segment WHERE a = b
+ """
+ sql """ set enable_expr_zonemap_filter = true """
+
+ // The single-slot path on the same table, to show that widening the
zonemap capability gate for
+ // two-slot shapes did not disturb the one-slot one it shares a gate with.
It has to be a
+ // predicate that expression zone maps actually see: `a = 'zzz'` is turned
into a ColumnPredicate
+ // and pruned by the ordinary olap path instead, which leaves
ExprZoneMapFilteredSegments at 0.
+ qt_literal_still_prunes """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_per_segment WHERE
starts_with(a, 'z')
+ """
+ def literalStillPrunesToken =
+ "expr_zonemap_pruning_per_segment_literal_" +
UUID.randomUUID().toString()
+ sql """
+ SELECT '${literalStillPrunesToken}', COUNT(*) FROM
test_expr_zonemap_pruning_per_segment
+ WHERE starts_with(a, 'z')
+ """
+ assertEquals(3L, filteredSegmentsOf(literalStillPrunesToken))
+
+ // And the ColumnPredicate shape still returns the right answer.
+ qt_column_predicate """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_per_segment WHERE a =
'zzz'
+ """
+
+ // Two DOUBLE columns on a native table. Unlike Parquet, the segment
writer records NaN presence
+ // in the zone map, so a float column with no NaN in it prunes normally.
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_two_columns_double
"""
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_two_columns_double (
+ id INT,
+ lo DOUBLE,
+ hi DOUBLE
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_two_columns_double
+ SELECT CAST(number AS INT), CAST(number AS DOUBLE), CAST(number +
10000 AS DOUBLE)
+ FROM numbers("number" = "4096")
+ """
+ sql """ sync """
+
+ qt_double_pruned """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_two_columns_double
+ WHERE lo > hi
+ """
+ def doubleToken = "expr_zonemap_pruning_two_columns_double_" +
UUID.randomUUID().toString()
+ sql """
+ SELECT '${doubleToken}', COUNT(*) FROM
test_expr_zonemap_pruning_two_columns_double
+ WHERE lo > hi
+ """
+ assertExprZonemapPruned(doubleToken)
+
+ // NaN and infinities make the bounds unusable for range pruning, so these
cases only have to
+ // return the right rows. has_nan / has_positive_inf / has_negative_inf
are what stops them.
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_two_columns_nan """
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_two_columns_nan (
+ id INT,
+ lo DOUBLE,
+ hi DOUBLE
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_two_columns_nan VALUES
+ (1, CAST('NaN' AS DOUBLE), 1.0),
+ (2, 1.0, CAST('NaN' AS DOUBLE)),
+ (3, CAST('infinity' AS DOUBLE), 1.0),
+ (4, CAST('-infinity' AS DOUBLE), 1.0),
+ (5, 1.0, 2.0),
+ (6, 2.0, 1.0)
+ """
+ sql """ sync """
+
+ def assertSameOnNanTable = { String predicate ->
+ sql """ set enable_expr_zonemap_filter = false """
+ def without = sql """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_two_columns_nan
WHERE ${predicate}
+ """
+ sql """ set enable_expr_zonemap_filter = true """
+ def with_ = sql """
+ SELECT COUNT(*) FROM test_expr_zonemap_pruning_two_columns_nan
WHERE ${predicate}
+ """
+ assertEquals(without[0][0] as long, with_[0][0] as long)
+ return with_[0][0] as long
+ }
+
+ // Doris orders NaN above every other value and treats NaN = NaN as true.
So lo > hi holds for
+ // rows 1, 3 and 6; lo < hi for rows 2, 4 and 5; and no row has the two
columns equal. The
+ // enabled-vs-disabled equality stays programmatic; the exact counts land
in golden output.
+ assertSameOnNanTable("lo > hi")
+ qt_nan_lo_gt_hi """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns_nan WHERE lo > hi"""
+ assertSameOnNanTable("lo != hi")
+ qt_nan_lo_ne_hi """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns_nan WHERE lo != hi"""
+ assertSameOnNanTable("lo = hi")
+ qt_nan_lo_eq_hi """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns_nan WHERE lo = hi"""
+ assertSameOnNanTable("lo < hi")
+ qt_nan_lo_lt_hi """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_two_columns_nan WHERE lo < hi"""
+
+ // FE compares two VARIANT paths without a cast, so `v['a'] = v['b']`
reaches the slot-slot gate
+ // as two VARIANT slots, while the segment types each path by its inferred
leaf type (INT here).
+ // The gate must reject VARIANT; admitting it fails the slot type check
and the query errors.
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_variant """
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_variant (id INT, v VARIANT)
+ DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ INSERT INTO test_expr_zonemap_pruning_variant VALUES
+ (1, parse_to_variant('{"a":1,"b":1}')), (2,
parse_to_variant('{"a":2,"b":3}')),
+ (3, parse_to_variant('{"a":5,"b":5}')), (4,
parse_to_variant('{"a":7,"b":8}'))
+ """
+ sql """ sync """
+ sql """ set enable_expr_zonemap_filter = true """
+ order_qt_variant_path_eq """SELECT id FROM
test_expr_zonemap_pruning_variant WHERE v['a'] = v['b']"""
+ order_qt_variant_path_ne """SELECT id FROM
test_expr_zonemap_pruning_variant WHERE v['a'] != v['b']"""
+
+ // __DORIS_VERSION_COL__ stores 0 on disk; a single-version segment reads
it through a
+ // ConstantColumnReader whose zone map is the rowset version. Expr
zone-map pruning must use that
+ // value, not the on-disk [0,0] placeholder.
+ sql """ set show_hidden_columns = true """
+ sql """ DROP TABLE IF EXISTS test_expr_zonemap_pruning_hidden_col """
+ sql """
+ CREATE TABLE test_expr_zonemap_pruning_hidden_col (
+ k INT,
+ v INT
+ ) UNIQUE KEY(k)
+ DISTRIBUTED BY HASH(k) BUCKETS 1
+ PROPERTIES (
+ "enable_unique_key_merge_on_write" = "true",
+ "replication_num" = "1",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sql """ INSERT INTO test_expr_zonemap_pruning_hidden_col SELECT number,
number FROM numbers("number" = "2048") """
+ sql """ sync """
+ def hiddenVersionRows = sql """ SELECT DISTINCT __DORIS_VERSION_COL__ FROM
test_expr_zonemap_pruning_hidden_col """
+ assertEquals(1, hiddenVersionRows.size())
+ long hiddenVersion = hiddenVersionRows[0][0] as long
+ assertTrue(hiddenVersion > 0)
+ def assertSameOnHiddenCol = { String predicate ->
+ sql """ set enable_expr_zonemap_filter = false """
+ def without = sql """ SELECT COUNT(*) FROM
test_expr_zonemap_pruning_hidden_col WHERE ${predicate} """
+ sql """ set enable_expr_zonemap_filter = true """
+ def with_ = sql """ SELECT COUNT(*) FROM
test_expr_zonemap_pruning_hidden_col WHERE ${predicate} """
+ assertEquals(without[0][0] as long, with_[0][0] as long)
+ return with_[0][0] as long
+ }
+ // Every row carries the version and `v = -1` matches nothing, so the
compound is the full 2048
+ // rows. Pruning against the [0,0] placeholder would drop the segment.
+ assertSameOnHiddenCol("__DORIS_VERSION_COL__ = ${hiddenVersion} OR v = -1")
+ qt_hidden_version_or_miss """SELECT COUNT(*) FROM
test_expr_zonemap_pruning_hidden_col WHERE __DORIS_VERSION_COL__ =
${hiddenVersion} OR v = -1"""
+ sql """ set show_hidden_columns = false """
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
