This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 9f7ce885ec5 [opt](search) Start a SEARCH conjunct from the rows the
scan's other filters kept (#70678)
9f7ce885ec5 is described below
commit 9f7ce885ec5db6431bd418de69a48c18e3f1806b
Author: guptashivam04 <[email protected]>
AuthorDate: Thu Nov 19 22:55:27 2026 +0530
[opt](search) Start a SEARCH conjunct from the rows the scan's other
filters kept (#70678)
### What problem does this PR solve?
Issue Number: None
Related PR: #70668 (split from it)
PR Type: Improvement
Problem Summary:
A SEARCH next to a selective filter outside it, as candidate search
writes it:
```sql
SELECT candidate_id FROM candidates
WHERE account_id = 12345
AND search('(title:"software engineer" OR title:"cloud engineer") AND
skill:ANY(python sql)')
```
evaluated the SEARCH over the whole segment, every account included, and
only then intersected the account's rows. `SegmentIterator` hides its
candidate rows from roots that combine three-valued bitmaps, and SEARCH
is one of them, so the rows `account_id = 12345` had already narrowed
the scan to were never seen by SEARCH.
### What is changed?
- A SEARCH that is a whole filtering conjunct gets the scan's candidate
rows at its root, through a new
`IndexQueryContext::search_root_candidates`. Its leaves still see none,
so how they combine three-valued results does not change. A SEARCH used
as a virtual column, or inside a compound predicate, is not affected.
- An unscored SEARCH collects its tree within those rows: a root that
lists its rows lists them within the candidates, any other root is read
through its scorer within them (`collect_truth_set` already handles both
sparse and dense candidates).
- The conjunct's result is only read within the candidates, so a
two-valued answer outside them is never used. A result computed this way
is partial, so it is never put into the DSL query cache.
- Scored SEARCH is unchanged: it keeps every match, and scores do not
depend on the candidates.
The candidate rows come from whatever narrowed the scan before SEARCH
runs: an inverted index on `account_id`, zone maps and the short key,
the delete bitmap.
### Numbers
2M rows, 3 BEs, inverted index on `account_id`, caches off, median of 3
runs. Rows and scores are byte-identical on 19 SEARCH queries (unscored
and scored, lucene and standard mode, top-k), with the inverted index
query cache off and on.
| query | before | after |
|---|---|---|
| ranking query of 16 SEARCH scans, account with 648 rows (0.03%) | 1648
ms, 7.9 s CPU | 1008 ms, 5.1 s CPU |
| same, account with 50K rows (2.5%) | 2002 ms, 8.8 s CPU | 1763 ms, 8.5
s CPU |
| `score()` queries (6 shapes, incl. top-k) | | unchanged |
A SEARCH whose root is an AND of OR groups gains little here, because
those groups still evaluate their children over the segment; a follow-up
lets nested groups read only the rows they are handed.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- `search/test_search_outer_filter_candidates`: on a table with an
inverted index on the account and on one with the account leading the
sort key, the rows of `account = 7 AND search(...)` against the full
SEARCH filtered afterwards, in lucene and standard mode, with AND, OR
and NOT; scores against the full SEARCH; with the query cache on, the
filtered query first, so a partial result would reach the unfiltered one
if it were cached.
- All `search/` and `inverted_index_p0` suites pass (298), except 3 that
fail the same way without this change on the same machine (custom
analyzer dictionaries and replica setup missing locally).
- [x] Unit Test
-
`BooleanTruthContractTest.CollectionWithinCandidatesKeepsTheirMatches`:
an unscored collection given candidates keeps exactly the matches among
them, for a weight that lists its rows and for one read through its
scorer.
- Existing suites pass (607 tests): SEARCH function and NULL handling,
BM25, compound predicate scores, segment iterator index application and
candidate pushdown, SNII readers, collectors.
- Behavior changed: No. Rows and scores are the same.
- Does this need documentation: No
---------
Co-authored-by: Shivam Gupta <[email protected]>
---
be/src/exprs/function/search/function_search.cpp | 11 +-
be/src/storage/index/index_query_context.h | 7 +
.../query/engine/collect/doc_set_collector.cpp | 10 +-
.../query/engine/collect/doc_set_collector.h | 3 +-
be/src/storage/segment/segment_iterator.cpp | 21 +-
.../query/engine/boolean_truth_contract_test.cpp | 94 +++
.../search/test_search_outer_filter_candidates.out | 741 +++++++++++++++++++++
.../test_search_outer_filter_candidates.groovy | 107 +++
8 files changed, 986 insertions(+), 8 deletions(-)
diff --git a/be/src/exprs/function/search/function_search.cpp
b/be/src/exprs/function/search/function_search.cpp
index 6d47098e5f6..e4e953c70dd 100644
--- a/be/src/exprs/function/search/function_search.cpp
+++ b/be/src/exprs/function/search/function_search.cpp
@@ -553,6 +553,12 @@ Status FunctionSearch::_evaluate_search_param(
top_k = index_query_context->query_limit;
}
+ // A SEARCH that is a whole conjunct reads only the rows the scan still
holds. Scored, it
+ // keeps every match, as the scores of the rows it returns do not depend
on them.
+ const roaring::Roaring* root_candidates = !enable_scoring &&
index_query_context != nullptr
+ ?
index_query_context->search_root_candidates
+ : nullptr;
+
auto weight = root_query->weight(enable_scoring);
if (!weight) {
LOG(WARNING) << "search: Failed to build query weight";
@@ -589,7 +595,7 @@ Status FunctionSearch::_evaluate_search_param(
query_v2::collect_multi_segment_doc_set(
weight, exec_ctx, root_binding_key, roaring,
index_query_context ?
index_query_context->collection_similarity : nullptr,
- enable_scoring);
+ enable_scoring, nullptr, root_candidates);
}
}
@@ -599,7 +605,8 @@ Status FunctionSearch::_evaluate_search_param(
bitmap_result =
InvertedIndexResultBitmap(std::move(roaring),
std::make_shared<roaring::Roaring>());
- if (dsl_cache && cache_usable) {
+ // A result read within candidates is partial and stays out of the cache.
+ if (dsl_cache && cache_usable && root_candidates == nullptr) {
insert_dsl_result(dsl_cache, seg_prefix, dsl_sig, bitmap_result);
}
diff --git a/be/src/storage/index/index_query_context.h
b/be/src/storage/index/index_query_context.h
index 32409627a76..52a360b8a3a 100644
--- a/be/src/storage/index/index_query_context.h
+++ b/be/src/storage/index/index_query_context.h
@@ -73,6 +73,13 @@ struct IndexQueryContext {
// evaluation.
const roaring::Roaring* candidate_rows = nullptr;
+ // The same candidate rows, for a SEARCH that is a whole conjunct on its
own. Its leaves
+ // combine three-valued bitmaps, so SegmentIterator hides candidate_rows
from them and hands
+ // the rows here instead: SEARCH starts its root from them, and its
two-valued result is
+ // read only within them. Set and reset like candidate_rows, and a result
computed from
+ // them is partial in the same way: it never goes into the query cache.
+ const roaring::Roaring* search_root_candidates = nullptr;
+
// ---- Reply direction: fields a READER writes and the CALLER reads back
----
//
// A reader writes them on the context it was given, never on a copy: a
copy's reply would
diff --git
a/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.cpp
b/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.cpp
index 3c9521081ea..51a4d5554f4 100644
--- a/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.cpp
+++ b/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.cpp
@@ -17,6 +17,7 @@
#include "storage/index/inverted/query/engine/collect/doc_set_collector.h"
+#include "common/check.h"
#include "storage/index/inverted/query/engine/complete_null_bitmap.h"
namespace doris::segment_v2::inverted_index::query_v2 {
@@ -25,12 +26,15 @@ void collect_multi_segment_doc_set(const WeightPtr& weight,
const QueryExecution
const std::string& binding_key,
const std::shared_ptr<roaring::Roaring>&
roaring,
const CollectionSimilarityPtr& similarity,
bool enable_scoring,
- roaring::Roaring* null_rows) {
+ roaring::Roaring* null_rows,
+ const roaring::Roaring* candidates) {
if (context.segment_num_rows == 0) {
return;
}
+ // Candidates restrict an unscored collection only; the caller reads its
rows within them.
+ DORIS_CHECK(candidates == nullptr || !enable_scoring);
if (!enable_scoring && weight->lists_rows(context, binding_key)) {
- auto rows = weight->listed_rows(context, binding_key, nullptr);
+ auto rows = weight->listed_rows(context, binding_key, candidates);
*roaring |= rows.true_rows;
if (null_rows != nullptr) {
*null_rows |= rows.null_rows;
@@ -49,6 +53,8 @@ void collect_multi_segment_doc_set(const WeightPtr& weight,
const QueryExecution
}
if (enable_scoring && similarity != nullptr) {
*roaring |= collect_scored_rows(scorer, 0, *similarity);
+ } else if (candidates != nullptr) {
+ *roaring |= collect_truth_set(scorer, context.null_resolver,
candidates).true_rows;
} else {
collect_true_rows(scorer, roaring.get());
}
diff --git
a/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.h
b/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.h
index ae26cba6429..dd36c92a5cb 100644
--- a/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.h
+++ b/be/src/storage/index/inverted/query/engine/collect/doc_set_collector.h
@@ -32,6 +32,7 @@ void collect_multi_segment_doc_set(const WeightPtr& weight,
const QueryExecution
const std::string& binding_key,
const std::shared_ptr<roaring::Roaring>&
roaring,
const CollectionSimilarityPtr& similarity,
bool enable_scoring,
- roaring::Roaring* null_rows = nullptr);
+ roaring::Roaring* null_rows = nullptr,
+ const roaring::Roaring* candidates =
nullptr);
} // namespace doris::segment_v2::inverted_index::query_v2
diff --git a/be/src/storage/segment/segment_iterator.cpp
b/be/src/storage/segment/segment_iterator.cpp
index 6edfbb7b18d..89f5777b067 100644
--- a/be/src/storage/segment/segment_iterator.cpp
+++ b/be/src/storage/segment/segment_iterator.cpp
@@ -909,6 +909,10 @@ static bool can_consume_candidate_rows(const
VExprContextSPtr& expr_ctx) {
DORIS_CHECK(virtual_expr != nullptr);
effective_root = virtual_expr.get();
}
+ // A filtering SEARCH starts from the candidate rows at its root.
+ if (!root->is_virtual_slot_ref() && root->node_type() ==
TExprNodeType::SEARCH_EXPR) {
+ return true;
+ }
if (effective_root->node_type() != TExprNodeType::MATCH_PRED) {
return false;
}
@@ -1505,7 +1509,10 @@ Status SegmentIterator::_apply_index_expr() {
// a compound predicate or a SEARCH, which combines its own leaf bitmaps
the
// same way, are therefore evaluated without the candidate; the top-level
// single-predicate consumption stays exact within the candidate.
- auto evaluate_with_candidate_policy = [&](const VExprContextSPtr&
expr_ctx) {
+ // A filtering conjunct that is a SEARCH by itself still gets the
candidate, at its root
+ // only: SEARCH answers two-valued, and the conjunct reads its result only
within the
+ // candidate.
+ auto evaluate_with_candidate_policy = [&](const VExprContextSPtr&
expr_ctx, bool filters) {
// Earlier expression conjuncts may have crossed the engage threshold.
// Refresh for both subsequent conjuncts and virtual-column
projections.
_refresh_candidate_pushdown();
@@ -1522,13 +1529,21 @@ Status SegmentIterator::_apply_index_expr() {
_index_query_context->candidate_rows != nullptr
&&
combines_index_results(*effective_root);
const roaring::Roaring* saved = suppress ?
_index_query_context->candidate_rows : nullptr;
+ const bool search_root = suppress && filters &&
!root->is_virtual_slot_ref() &&
+ root->node_type() ==
TExprNodeType::SEARCH_EXPR;
if (suppress) {
_index_query_context->candidate_rows = nullptr;
}
+ if (search_root) {
+ _index_query_context->search_root_candidates = saved;
+ }
Status st = expr_ctx->evaluate_inverted_index(num_rows());
if (suppress) {
_index_query_context->candidate_rows = saved;
}
+ if (search_root) {
+ _index_query_context->search_root_candidates = nullptr;
+ }
return st;
};
@@ -1555,7 +1570,7 @@ Status SegmentIterator::_apply_index_expr() {
// answer approximately then knows its work will be used. The virtual
column loop
// further down consumes none, and deliberately names none.
expr_ctx->set_approx_index_result_consumer(expr_ctx->root().get());
- if (Status st = evaluate_with_candidate_policy(expr_ctx); !st.ok()) {
+ if (Status st = evaluate_with_candidate_policy(expr_ctx,
/*filters=*/true); !st.ok()) {
if (_downgrade_without_index(st) || st.code() ==
ErrorCode::NOT_IMPLEMENTED_ERROR) {
continue;
} else {
@@ -1605,7 +1620,7 @@ Status SegmentIterator::_apply_index_expr() {
if (expr_ctx->get_index_context() == nullptr) {
continue;
}
- if (Status st = evaluate_with_candidate_policy(expr_ctx);
!st.ok()) {
+ if (Status st = evaluate_with_candidate_policy(expr_ctx,
/*filters=*/false); !st.ok()) {
if (_downgrade_without_index(st) || st.code() ==
ErrorCode::NOT_IMPLEMENTED_ERROR) {
continue;
} else {
diff --git
a/be/test/storage/index/inverted/query/engine/boolean_truth_contract_test.cpp
b/be/test/storage/index/inverted/query/engine/boolean_truth_contract_test.cpp
index 41503b945bd..acff6829085 100644
---
a/be/test/storage/index/inverted/query/engine/boolean_truth_contract_test.cpp
+++
b/be/test/storage/index/inverted/query/engine/boolean_truth_contract_test.cpp
@@ -30,6 +30,7 @@
#include "storage/index/inverted/query/engine/bit_set_query/bit_set_scorer.h"
#include
"storage/index/inverted/query/engine/boolean_query/boolean_query_builder.h"
#include "storage/index/inverted/query/engine/boolean_query/operator.h"
+#include "storage/index/inverted/query/engine/collect/doc_set_collector.h"
#include "storage/index/inverted/query/engine/null_bitmap_test_util.h"
namespace doris::segment_v2::inverted_index::query_v2 {
@@ -559,5 +560,98 @@ TEST(BooleanTruthContractTest,
OptionalShouldBesideARequiredClauseIsTwoValued) {
}
}
+// Lists its rows itself, like a phrase root, and records the candidates each
listing gets.
+class ListingWeight final : public Weight {
+public:
+ ListingWeight(TruthRows rows,
std::shared_ptr<std::vector<roaring::Roaring>> listed)
+ : _rows(std::move(rows)), _listed(std::move(listed)) {}
+ ScorerPtr scorer(const QueryExecutionContext& /*context*/) override {
+ ADD_FAILURE() << "a weight that lists its rows is not read through a
scorer";
+ return nullptr;
+ }
+ bool lists_rows(const QueryExecutionContext& /*context*/,
+ const std::string& /*binding_key*/) const override {
+ return true;
+ }
+ index_query::TruthSet listed_rows(const QueryExecutionContext& /*context*/,
+ const std::string& /*binding_key*/,
+ const roaring::Roaring* candidates)
override {
+ EXPECT_NE(candidates, nullptr);
+ index_query::TruthSet result {*_rows.true_rows, *_rows.null_rows};
+ if (candidates != nullptr) {
+ _listed->push_back(*candidates);
+ result.true_rows &= *candidates;
+ result.null_rows &= *candidates;
+ }
+ return result;
+ }
+
+private:
+ TruthRows _rows;
+ std::shared_ptr<std::vector<roaring::Roaring>> _listed;
+};
+
+class ListingQuery final : public Query {
+public:
+ ListingQuery(TruthRows rows,
std::shared_ptr<std::vector<roaring::Roaring>> listed)
+ : _rows(std::move(rows)), _listed(std::move(listed)) {}
+ WeightPtr weight(bool /*enable_scoring*/) override {
+ return std::make_shared<ListingWeight>(_rows, _listed);
+ }
+
+private:
+ TruthRows _rows;
+ std::shared_ptr<std::vector<roaring::Roaring>> _listed;
+};
+
+// An unscored collection given candidates keeps exactly the matches among
them. A root that
+// lists its rows lists them within the candidates; any other root is read
through its scorer,
+// seeking candidate by candidate when they are sparse and intersecting once
when they are dense.
+TEST(BooleanTruthContractTest, CollectionWithinCandidatesKeepsTheirMatches) {
+ constexpr uint32_t row_count = 4096;
+ TruthRows rows;
+ for (uint32_t doc = 0; doc < row_count; doc += 3) {
+ rows.true_rows->add(doc);
+ }
+ roaring::Roaring sparse;
+ for (uint32_t doc = 0; doc < row_count; doc += 97) {
+ sparse.add(doc);
+ }
+ roaring::Roaring dense;
+ for (uint32_t doc = 0; doc < row_count; doc += 2) {
+ dense.add(doc);
+ }
+ auto collect = [&](const QueryPtr& query, const roaring::Roaring&
candidates) {
+ QueryExecutionContext context;
+ context.segment_num_rows = row_count;
+ auto collected = std::make_shared<roaring::Roaring>();
+ collect_multi_segment_doc_set(query->weight(false), context, "",
collected, nullptr,
+ /*enable_scoring=*/false, nullptr,
&candidates);
+ return *collected;
+ };
+
+ for (const auto* candidates : {&sparse, &dense}) {
+ auto listed = std::make_shared<std::vector<roaring::Roaring>>();
+ EXPECT_EQ(collect(std::make_shared<ListingQuery>(rows, listed),
*candidates),
+ *rows.true_rows & *candidates);
+ ASSERT_EQ(listed->size(), 1U);
+ EXPECT_EQ(listed->front(), *candidates);
+ }
+
+ // Sparse candidates: the scorer only seeks to them, and never walks the
rows between.
+ auto work = std::make_shared<ScorerWork>();
+ EXPECT_EQ(collect(std::make_shared<ForwardOnlyQuery>(rows, 1.0F, work),
sparse),
+ *rows.true_rows & sparse);
+ EXPECT_GT(work->seeks, 0U);
+ EXPECT_LE(work->seeks, sparse.cardinality());
+ EXPECT_EQ(work->advances, 0U);
+
+ // Dense candidates: the scorer is read once and intersected with them.
+ work = std::make_shared<ScorerWork>();
+ EXPECT_EQ(collect(std::make_shared<ForwardOnlyQuery>(rows, 1.0F, work),
dense),
+ *rows.true_rows & dense);
+ EXPECT_EQ(work->seeks, 0U);
+}
+
} // namespace
} // namespace doris::segment_v2::inverted_index::query_v2
diff --git
a/regression-test/data/search/test_search_outer_filter_candidates.out
b/regression-test/data/search/test_search_outer_filter_candidates.out
new file mode 100644
index 00000000000..dcdc8370c30
--- /dev/null
+++ b/regression-test/data/search/test_search_outer_filter_candidates.out
@@ -0,0 +1,741 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !idx_lucene_0_rows --
+136
+20
+576
+580
+620
+796
+880
+956
+
+-- !idx_lucene_0_scores --
+136 6.1794
+20 5.7438
+576 6.7006
+580 5.7438
+620 5.7438
+796 6.1794
+880 5.7438
+956 6.1794
+
+-- !idx_lucene_0_cached_filtered --
+8
+
+-- !idx_lucene_0_cached_full --
+92
+
+-- !idx_lucene_0_cached_filtered_again --
+8
+
+-- !idx_standard_0_rows --
+136
+20
+576
+580
+620
+796
+880
+956
+
+-- !idx_standard_0_scores --
+136 6.1794
+20 5.7438
+576 6.7006
+580 5.7438
+620 5.7438
+796 6.1794
+880 5.7438
+956 6.1794
+
+-- !idx_standard_0_cached_filtered --
+8
+
+-- !idx_standard_0_cached_full --
+92
+
+-- !idx_standard_0_cached_filtered_again --
+8
+
+-- !idx_lucene_1_rows --
+20
+294
+354
+454
+580
+614
+620
+880
+
+-- !idx_lucene_1_scores --
+20 2.8998
+294 3.4113
+354 3.4113
+454 2.8909
+580 2.8998
+614 2.8909
+620 2.8998
+880 2.8998
+
+-- !idx_lucene_1_cached_filtered --
+8
+
+-- !idx_lucene_1_cached_full --
+93
+
+-- !idx_lucene_1_cached_filtered_again --
+8
+
+-- !idx_standard_1_rows --
+20
+294
+354
+454
+580
+614
+620
+880
+
+-- !idx_standard_1_scores --
+20 2.8998
+294 3.4113
+354 3.4113
+454 2.8909
+580 2.8998
+614 2.8909
+620 2.8998
+880 2.8998
+
+-- !idx_standard_1_cached_filtered --
+8
+
+-- !idx_standard_1_cached_full --
+93
+
+-- !idx_standard_1_cached_filtered_again --
+8
+
+-- !idx_lucene_2_rows --
+136
+144
+175
+20
+224
+232
+324
+348
+372
+384
+415
+448
+45
+508
+565
+575
+576
+580
+620
+628
+752
+785
+796
+824
+84
+880
+956
+975
+
+-- !idx_lucene_2_scores --
+144 1.5904
+20 2.8065
+324 1.5904
+348 1.5904
+372 1.5904
+384 1.5904
+580 2.8065
+620 2.8065
+84 1.5904
+880 2.8065
+
+-- !idx_lucene_2_cached_filtered --
+28
+
+-- !idx_lucene_2_cached_full --
+334
+
+-- !idx_lucene_2_cached_filtered_again --
+28
+
+-- !idx_standard_2_rows --
+136
+144
+175
+20
+224
+232
+324
+348
+372
+384
+415
+448
+45
+508
+565
+575
+576
+580
+620
+628
+752
+785
+796
+824
+84
+880
+956
+975
+
+-- !idx_standard_2_scores --
+144 1.5904
+20 2.8065
+324 1.5904
+348 1.5904
+372 1.5904
+384 1.5904
+580 2.8065
+620 2.8065
+84 1.5904
+880 2.8065
+
+-- !idx_standard_2_cached_filtered --
+28
+
+-- !idx_standard_2_cached_full --
+334
+
+-- !idx_standard_2_cached_filtered_again --
+28
+
+-- !idx_lucene_3_rows --
+144
+224
+289
+294
+324
+354
+384
+454
+479
+614
+679
+699
+719
+754
+824
+829
+84
+949
+999
+
+-- !idx_lucene_3_scores --
+144 2.9175
+224 2.9175
+289 2.9175
+294 2.9175
+324 2.9175
+354 2.9175
+384 2.9175
+454 2.9175
+479 2.9175
+84 2.9175
+
+-- !idx_lucene_3_cached_filtered --
+19
+
+-- !idx_lucene_3_cached_full --
+200
+
+-- !idx_lucene_3_cached_filtered_again --
+19
+
+-- !idx_standard_3_rows --
+144
+224
+289
+294
+324
+354
+384
+454
+479
+614
+679
+699
+719
+754
+824
+829
+84
+949
+999
+
+-- !idx_standard_3_scores --
+144 2.9175
+224 2.9175
+289 2.9175
+294 2.9175
+324 2.9175
+354 2.9175
+384 2.9175
+454 2.9175
+479 2.9175
+84 2.9175
+
+-- !idx_standard_3_cached_filtered --
+19
+
+-- !idx_standard_3_cached_full --
+200
+
+-- !idx_standard_3_cached_filtered_again --
+19
+
+-- !idx_lucene_4_rows --
+136
+576
+796
+956
+
+-- !idx_lucene_4_scores --
+136 2.8948
+576 3.416
+796 2.8948
+956 2.8948
+
+-- !idx_lucene_4_cached_filtered --
+4
+
+-- !idx_lucene_4_cached_full --
+46
+
+-- !idx_lucene_4_cached_filtered_again --
+4
+
+-- !idx_standard_4_rows --
+136
+576
+796
+956
+
+-- !idx_standard_4_scores --
+136 2.8948
+576 3.416
+796 2.8948
+956 2.8948
+
+-- !idx_standard_4_cached_filtered --
+4
+
+-- !idx_standard_4_cached_full --
+46
+
+-- !idx_standard_4_cached_filtered_again --
+4
+
+-- !idx_lucene_5_rows --
+121
+136
+221
+26
+351
+421
+511
+521
+546
+576
+601
+611
+621
+701
+771
+796
+806
+811
+831
+86
+956
+
+-- !idx_lucene_5_scores --
+121 1.8993
+136 1.8993
+221 1.8993
+26 1.8993
+351 1.8993
+421 1.8993
+511 1.8993
+521 1.8993
+546 1.8993
+86 1.8993
+
+-- !idx_lucene_5_cached_filtered --
+21
+
+-- !idx_lucene_5_cached_full --
+200
+
+-- !idx_lucene_5_cached_filtered_again --
+21
+
+-- !idx_standard_5_rows --
+121
+136
+221
+26
+351
+421
+511
+521
+546
+576
+601
+611
+621
+701
+771
+796
+806
+811
+831
+86
+956
+
+-- !idx_standard_5_scores --
+121 1.8993
+136 1.8993
+221 1.8993
+26 1.8993
+351 1.8993
+421 1.8993
+511 1.8993
+521 1.8993
+546 1.8993
+86 1.8993
+
+-- !idx_standard_5_cached_filtered --
+21
+
+-- !idx_standard_5_cached_full --
+200
+
+-- !idx_standard_5_cached_filtered_again --
+21
+
+-- !key_lucene_0_rows --
+136
+20
+576
+580
+620
+796
+880
+956
+
+-- !key_lucene_0_cached_filtered --
+8
+
+-- !key_lucene_0_cached_full --
+92
+
+-- !key_lucene_0_cached_filtered_again --
+8
+
+-- !key_standard_0_rows --
+136
+20
+576
+580
+620
+796
+880
+956
+
+-- !key_standard_0_cached_filtered --
+8
+
+-- !key_standard_0_cached_full --
+92
+
+-- !key_standard_0_cached_filtered_again --
+8
+
+-- !key_lucene_1_rows --
+20
+294
+354
+454
+580
+614
+620
+880
+
+-- !key_lucene_1_cached_filtered --
+8
+
+-- !key_lucene_1_cached_full --
+93
+
+-- !key_lucene_1_cached_filtered_again --
+8
+
+-- !key_standard_1_rows --
+20
+294
+354
+454
+580
+614
+620
+880
+
+-- !key_standard_1_cached_filtered --
+8
+
+-- !key_standard_1_cached_full --
+93
+
+-- !key_standard_1_cached_filtered_again --
+8
+
+-- !key_lucene_2_rows --
+136
+144
+175
+20
+224
+232
+324
+348
+372
+384
+415
+448
+45
+508
+565
+575
+576
+580
+620
+628
+752
+785
+796
+824
+84
+880
+956
+975
+
+-- !key_lucene_2_cached_filtered --
+28
+
+-- !key_lucene_2_cached_full --
+334
+
+-- !key_lucene_2_cached_filtered_again --
+28
+
+-- !key_standard_2_rows --
+136
+144
+175
+20
+224
+232
+324
+348
+372
+384
+415
+448
+45
+508
+565
+575
+576
+580
+620
+628
+752
+785
+796
+824
+84
+880
+956
+975
+
+-- !key_standard_2_cached_filtered --
+28
+
+-- !key_standard_2_cached_full --
+334
+
+-- !key_standard_2_cached_filtered_again --
+28
+
+-- !key_lucene_3_rows --
+144
+224
+289
+294
+324
+354
+384
+454
+479
+614
+679
+699
+719
+754
+824
+829
+84
+949
+999
+
+-- !key_lucene_3_cached_filtered --
+19
+
+-- !key_lucene_3_cached_full --
+200
+
+-- !key_lucene_3_cached_filtered_again --
+19
+
+-- !key_standard_3_rows --
+144
+224
+289
+294
+324
+354
+384
+454
+479
+614
+679
+699
+719
+754
+824
+829
+84
+949
+999
+
+-- !key_standard_3_cached_filtered --
+19
+
+-- !key_standard_3_cached_full --
+200
+
+-- !key_standard_3_cached_filtered_again --
+19
+
+-- !key_lucene_4_rows --
+136
+576
+796
+956
+
+-- !key_lucene_4_cached_filtered --
+4
+
+-- !key_lucene_4_cached_full --
+46
+
+-- !key_lucene_4_cached_filtered_again --
+4
+
+-- !key_standard_4_rows --
+136
+576
+796
+956
+
+-- !key_standard_4_cached_filtered --
+4
+
+-- !key_standard_4_cached_full --
+46
+
+-- !key_standard_4_cached_filtered_again --
+4
+
+-- !key_lucene_5_rows --
+121
+136
+221
+26
+351
+421
+511
+521
+546
+576
+601
+611
+621
+701
+771
+796
+806
+811
+831
+86
+956
+
+-- !key_lucene_5_cached_filtered --
+21
+
+-- !key_lucene_5_cached_full --
+200
+
+-- !key_lucene_5_cached_filtered_again --
+21
+
+-- !key_standard_5_rows --
+121
+136
+221
+26
+351
+421
+511
+521
+546
+576
+601
+611
+621
+701
+771
+796
+806
+811
+831
+86
+956
+
+-- !key_standard_5_cached_filtered --
+21
+
+-- !key_standard_5_cached_full --
+200
+
+-- !key_standard_5_cached_filtered_again --
+21
+
diff --git
a/regression-test/suites/search/test_search_outer_filter_candidates.groovy
b/regression-test/suites/search/test_search_outer_filter_candidates.groovy
new file mode 100644
index 00000000000..f2af20b714f
--- /dev/null
+++ b/regression-test/suites/search/test_search_outer_filter_candidates.groovy
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// A SEARCH that is a whole conjunct starts from the rows the scan's other
filters kept, read
+// through an inverted index or a sort key. The expected rows and scores in
the .out file are
+// those of the SEARCH over the whole segment, filtered afterwards; with the
query cache on, a
+// partial result must not reach a later query without the filter.
+suite("test_search_outer_filter_candidates") {
+ def tables = [
+ idx: "test_search_outer_filter_idx",
+ key: "test_search_outer_filter_key",
+ ]
+ def textIndex = 'USING INVERTED PROPERTIES("parser" = "unicode",
"lower_case" = "true", "support_phrase" = "true")'
+ tables.each { kind, table ->
+ sql "DROP TABLE IF EXISTS ${table}"
+ def columns = kind == "idx" ? "id INT, account INT" : "account INT NOT
NULL, id INT"
+ def accountIndex = kind == "idx" ? ", INDEX idx_account (account)
USING INVERTED" : ""
+ sql """
+ CREATE TABLE ${table} (
+ ${columns},
+ title TEXT,
+ skill TEXT,
+ place TEXT,
+ INDEX idx_title (title) ${textIndex},
+ INDEX idx_skill (skill) ${textIndex},
+ INDEX idx_place (place) ${textIndex}
+ ${accountIndex}
+ ) ENGINE=OLAP
+ DUPLICATE KEY(${kind == "idx" ? "id" : "account, id"})
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1")
+ """
+ // Account 7 holds about a tenth of the rows, spread by a hash so its
titles, skills and
+ // places vary like the others'; NULL titles on every 13th row.
+ sql """
+ INSERT INTO ${table} (id, account, title, skill, place)
+ SELECT number,
+ if(abs(murmur_hash3_32(concat('account-', number))) % 10 =
0, 7, number % 50 + 100),
+ if(number % 13 = 0, NULL, concat_ws(' ',
+ element_at(['', 'senior', 'lead'], number % 3 + 1),
+ element_at(['software', 'cloud', 'sales', 'data',
'ai'], number % 5 + 1),
+ element_at(['engineer', 'manager', 'developer',
'analyst'], number % 4 + 1))),
+ element_at(['python sql', 'docker', 'team leadership',
'excel', 'project management'],
+ number % 5 + 1),
+ element_at(['Manila PH', 'Cebu Philippines', 'Berlin
Germany', 'Pune India'],
+ number % 4 + 1)
+ FROM numbers("number" = "1000")
+ """
+ }
+ sql "sync"
+ sql "set enable_common_expr_pushdown = true"
+
+ def dsls = [
+ // AND of OR groups, as candidate search writes them
+ '(skill:ANY(python docker excel) OR skill:"team leadership") AND ' +
+ '(title:"software engineer" OR title:"cloud engineer" OR
title:"data analyst") AND ' +
+ '(place:ANY(ph pilipinas) OR place:"cebu philippines")',
+ 'title:"software engineer" OR title:"ai developer"',
+ '(title:ANY(engineer developer) OR skill:sql) AND NOT place:berlin',
+ 'skill:"project management" AND NOT title:"sales manager"',
+ // a phrase root, which lists its rows itself
+ 'title:"cloud engineer"',
+ // a term root, read through its scorer
+ 'skill:docker',
+ ]
+ def modes = [lucene: '{"mode":"lucene"}', standard: '{"mode":"standard"}']
+
+ tables.each { kind, table ->
+ dsls.eachWithIndex { dsl, i ->
+ modes.each { modeName, mode ->
+ def search = "search('${dsl}', '${mode}')"
+ def tag = "${kind}_${modeName}_${i}"
+ sql "set enable_inverted_index_query_cache = false"
+ "order_qt_${tag}_rows"("SELECT id FROM ${table} WHERE account
= 7 AND ${search}")
+ if (kind == "idx") {
+ "order_qt_${tag}_scores"("""
+ SELECT id, ROUND(score(), 4) AS s FROM ${table} WHERE
account = 7 AND ${search}
+ ORDER BY score() DESC, id LIMIT 10""")
+ }
+
+ // With the cache on, the filtered query first: the query
without the filter
+ // must still count every row of the SEARCH.
+ sql "set enable_inverted_index_query_cache = true"
+ "qt_${tag}_cached_filtered"("SELECT count(*) FROM ${table}
WHERE account = 7 AND ${search}")
+ "qt_${tag}_cached_full"("SELECT count(*) FROM ${table} WHERE
${search}")
+ "qt_${tag}_cached_filtered_again"("SELECT count(*) FROM
${table} WHERE account = 7 AND ${search}")
+ }
+ }
+ }
+ sql "set enable_inverted_index_query_cache = false"
+
+ tables.each { kind, table -> sql "DROP TABLE IF EXISTS ${table}" }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]