guptashivam04 opened a new pull request, #70668:
URL: https://github.com/apache/doris/pull/70668
### What problem does this PR solve?
Issue Number: None
Related PR: #70488 (same candidate-search query, numeric SEARCH leaves)
PR Type: Improvement
Problem Summary:
Candidate-search queries look like this, run without `score()`:
```sql
SELECT candidate_id FROM candidates
WHERE account_id = 12345
AND search('(title:"software engineer" OR ... 30 phrases on 2 fields)
AND (skill:ANY(python sql ...) OR skill:"solution
architecture" OR ... 46 phrases)
AND (country:ANY(ph phl) OR country:"republic philippines" OR
... 5 fields)')
```
Even though only a small share of the rows can match, each SEARCH did work
in proportion to the whole segment:
- **`account_id = X` outside SEARCH did not narrow it.** The scan hides its
candidate rows from compound and SEARCH roots, so SEARCH evaluated every
account and the scan intersected afterwards.
- **Every group was checked over the whole segment.** In lucene mode the AND
is a doc-at-a-time intersection over `BufferedUnion`s, which refill a ~4k-row
window from every child, so phrase children checked positions on rows the other
groups had already ruled out. A nested OR in standard mode built its full
three-valued bitmap first.
- **Phrases sharing words read them again.** Each phrase query opened,
decoded and intersected its own copy of every word.
- **Groups ran in the order written,** so a broad group written first was
read in full before a rare one could narrow it.
- **The scan read text the index had already answered.** Variant subcolumns
share their variant's unique id in the output set, so a long text subcolumn
used only by an index-answered filter was still read for every matching row.
### What is changed?
Five commits; each message has the details.
1. **Phrase sets.** Two or more phrases of one field under an unscored OR
run as one set: each distinct word is opened once, a row is a candidate when it
holds all words of some phrase, and positions are read once per word and row,
only on candidates. Bounded at 32 MB of document lists per segment; past that,
and for scored queries and sources that batch reads, the phrase queries run as
before.
2. **Nested groups read only the kept rows.** Unscored `OccurBooleanWeight`
and `OperatorBooleanWeight` list their rows within the candidates an AND hands
them: required clauses first, OR children only on rows not yet matched, NOT
within the candidates. Lucene mode stays two-valued, standard mode keeps its
UNKNOWN rows, and `minimum_should_match > 1` keeps the scorer path.
3. **Rarest first.** Weights estimate their rows from the term dictionary
without reading postings, and conjunctions read their row-listing clauses from
the smallest estimate up.
4. **Outer filter rows.** A SEARCH that is a whole filtering conjunct starts
from the rows the scan's other filters kept (an inverted index on `account_id`,
key ranges, zone maps, the delete bitmap). Only the root reads within them; a
result read this way never enters the DSL cache; scored SEARCH is unchanged.
5. **Variant read skip.** A variant subcolumn read only by an index-answered
filter is not read when no column of that variant is output and the scan has no
virtual columns. A conjunct that is the column itself or a cast of it now
counts as one the index does not answer, which also fixes `v['fs'] MATCH 'true'
AND CAST(v['fs'] AS BOOLEAN)` returning no rows with the index on.
### Numbers
2M candidates, 3 BEs, one account at 2.5% of rows, caches off, median of 3
runs, before = base, after = this PR. Rows and scores returned are
byte-identical in every case (19 queries, with the inverted index query cache
off and on).
| query | before | after |
|---|---|---|
| full ranking query (16 SEARCH scans), account at 2.5% | 2057 ms, 10.4 s
CPU, 3.8 GB scanned | 516 ms, 1.8 s CPU, 375 MB |
| same, account at 0.03% | 1682 ms, 9.1 s CPU | 189 ms, 0.7 s CPU |
| one SEARCH, 3 AND'ed groups, `account_id` outside | 248 ms | 38 ms |
| `resume_search`, 40 phrases + 30 words, count | 668 ms | 17 ms |
| `score()` queries (6 shapes, incl. top-k) | 45–685 ms | unchanged |
| 100 accounts submitting the full query at once | 217 s, 706 s BE CPU | 31
s, 82 s BE CPU |
### Release note
SEARCH without `score()` reads only the rows the other conditions of the
query can still match: an outer filter such as `account_id = X`, and the other
groups of an AND. OR'ed phrases of one field share their word reads, and long
variant text answered by the index is no longer read.
### Check List (For Author)
- Test
- [x] Unit Test
- `phrase_set_test`: the set against each phrase run alone on 20
random indexes, with and without candidates, seeks, NULL rows on both source
kinds, one open per common word for a few candidates.
- `boolean_truth_contract_test`: nested AND/OR/NOT within sparse and
dense candidates against a scalar three-valued oracle; occur Booleans list
exactly the scorer's rows on random trees; rarest-first ordering; collection
within candidates.
- Existing suites pass (342 tests in 17 suites, 2 skipped by the
build's feature flags): `FunctionSearchTest`, `FunctionSearchNullTest`,
`VCompoundPredScoreTest`, `BM25SimilarityTest`, `ListedTermsTest`,
`SniiIndexReader*`, `SegmentIteratorApplyIndexExprTest`,
`SegmentIteratorCandidatePushdownTest`, `CountEmitShortcut`,
`InvertedIndexArrayTest`, `VariantColumnWriterReaderTest`,
`MultiSegmentCollectorTest`, `DocSetCollectorTest`.
- [x] Regression test
- `search/test_search_phrase_sets`: rows of OR'ed phrases (phrases
sharing words in either order, a word no row holds, NULL rows, text and variant
fields, V2 and SNII, lucene and standard modes) against each phrase run alone
and against MATCH_PHRASE; `score()` against the sum of the phrases' scores, at
every row and on the top-k path; inside AND and AND NOT.
- `search/test_search_outer_filter_candidates`: rows against the
full SEARCH filtered afterwards, scores unchanged, the cache never hands a
partial result to an unfiltered query; index and sort-key tables.
- `variant_p0/test_variant_index_filter_only_read`: selected values
(the filtered subcolumn, a sibling, the whole variant, nothing) against the
index off and against a read by id; subcolumns also used by expressions no
index answers.
- All 46 suites under `search/` pass, and `variant_p0` plus
`inverted_index_p0` (467 suites) fail only the same 6 suites as the base build
on the same machine (they depend on local setup: custom analyzer dictionaries,
outfile, replicas).
- Behavior changed: No. The rows and scores returned are the same; one wrong
result is fixed (point 5).
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]