guptashivam04 opened a new pull request, #70678:
URL: https://github.com/apache/doris/pull/70678
### What problem does this PR solve?
Issue Number: None
Related PR: #70668 (split from it)
PR Type: Improvement
Problem Summary:
A SEARCH next to a selective filter outside it, as candidate search writes
it:
```sql
SELECT candidate_id FROM candidates
WHERE account_id = 12345
AND search('(title:"software engineer" OR title:"cloud engineer") AND
skill:ANY(python sql)')
```
evaluated the SEARCH over the whole segment, every account included, and
only then intersected the account's rows. `SegmentIterator` hides its candidate
rows from roots that combine three-valued bitmaps, and SEARCH is one of them,
so the rows `account_id = 12345` had already narrowed the scan to were never
seen by SEARCH.
### What is changed?
- A SEARCH that is a whole filtering conjunct gets the scan's candidate rows
at its root, through a new `IndexQueryContext::search_root_candidates`. Its
leaves still see none, so how they combine three-valued results does not
change. A SEARCH used as a virtual column, or inside a compound predicate, is
not affected.
- An unscored SEARCH collects its tree within those rows: a root that lists
its rows lists them within the candidates, any other root is read through its
scorer within them (`collect_truth_set` already handles both sparse and dense
candidates).
- The conjunct's result is only read within the candidates, so a two-valued
answer outside them is never used. A result computed this way is partial, so it
is never put into the DSL query cache.
- Scored SEARCH is unchanged: it keeps every match, and scores do not depend
on the candidates.
The candidate rows come from whatever narrowed the scan before SEARCH runs:
an inverted index on `account_id`, zone maps and the short key, the delete
bitmap.
### Numbers
2M rows, 3 BEs, inverted index on `account_id`, caches off, median of 3
runs. Rows and scores are byte-identical on 19 SEARCH queries (unscored and
scored, lucene and standard mode, top-k), with the inverted index query cache
off and on.
| query | before | after |
|---|---|---|
| ranking query of 16 SEARCH scans, account with 648 rows (0.03%) | 1648 ms,
7.9 s CPU | 1008 ms, 5.1 s CPU |
| same, account with 50K rows (2.5%) | 2002 ms, 8.8 s CPU | 1763 ms, 8.5 s
CPU |
| `score()` queries (6 shapes, incl. top-k) | | unchanged |
A SEARCH whose root is an AND of OR groups gains little here, because those
groups still evaluate their children over the segment; a follow-up lets nested
groups read only the rows they are handed.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- `search/test_search_outer_filter_candidates`: on a table with an
inverted index on the account and on one with the account leading the sort key,
the rows of `account = 7 AND search(...)` against the full SEARCH filtered
afterwards, in lucene and standard mode, with AND, OR and NOT; scores against
the full SEARCH; with the query cache on, the filtered query first, so a
partial result would reach the unfiltered one if it were cached.
- All `search/` and `inverted_index_p0` suites pass (298), except 3
that fail the same way without this change on the same machine (custom analyzer
dictionaries and replica setup missing locally).
- [x] Unit Test
-
`BooleanTruthContractTest.CollectionWithinCandidatesKeepsTheirMatches`: an
unscored collection given candidates keeps exactly the matches among them, for
a weight that lists its rows and for one read through its scorer.
- Existing suites pass (607 tests): SEARCH function and NULL
handling, BM25, compound predicate scores, segment iterator index application
and candidate pushdown, SNII readers, collectors.
- Behavior changed: No. Rows and scores are the same.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]