airborne12 opened a new pull request, #69791:
URL: https://github.com/apache/doris/pull/69791
### What problem does this PR solve?
Issue Number: None
Related PR: #69790
Problem Summary: score() of a MATCH on an untokenized (keyword) inverted
index is always 0, while SEARCH on the same field scores BM25:
INDEX idx_redirect (redirect) USING INVERTED
-- one row ('BP')
SELECT score() FROM t WHERE redirect MATCH 'BP'; -- 0
SELECT score() FROM t WHERE SEARCH('redirect:BP'); -- 0.2876821
Elasticsearch scores a match, term or match_phrase query on a keyword
field with BM25 in query context: no norms and no frequencies, so
freq = dl = avgdl = 1 and the score is the idf of the whole value
(0.2876821 above, 0.6931471 for N=4, n=2). Only a bool filter scores 0.
SEARCH already gives exactly these values; MATCH on keyword behaved like
a filter, so `body MATCH 'apple' AND tag MATCH 'x'` ranked as if the tag
clause were not there.
Root cause on branch-4.1, three places keep keyword MATCH out of scoring:
StringTypeInvertedIndexReader::query never sets is_similarity_score (and
answers score() queries from the query cache), MatchPredicateCollector
skips indexes without an analyzer, and is_need_similarity_score only
accepts phrase-capable or lowercase-normalizer indexes.
Fix: a keyword (non-analyzed) index is eligible for MATCH scoring
regardless of support_phrase; the keyword reader requests scoring for
the MATCH query types and bypasses the result cache like the full-text
reader does; MatchPredicateCollector collects the whole value as one
term (case and spaces kept), the term the reader looks up. The legacy
scorer already uses norm 1 when a field has none. Equality runs as
MATCH_ANY inside the reader but keeps EQUAL_QUERY for the decision, so
it stays a pure filter. MATCH_REGEXP and MATCH_PHRASE_PREFIX keep their
current scores.
### Release note
score() of MATCH / MATCH_ALL / MATCH_PHRASE on a keyword (untokenized)
inverted index now returns the BM25 score of the value, the same as
SEARCH and Elasticsearch, instead of 0.
### Check List (For Author)
- Test: Regression test / Unit Test / Manual test
- Regression test: new test_match_keyword_score (V2/V3, implicit and
explicit parser=none); its output equals the master PR's V2/V3
blocks
- Unit Test: IndexReaderHelperTest, CollectionStatisticsTest
- Manual test: the reported table on a branch-4.1 ASAN build; MATCH
equals SEARCH and Elasticsearch 9.5.5 to 7 digits, also for
ARRAY<STRING>, a column with both a keyword and an analyzed index,
and OR/AND mixes
- Behavior changed: Yes. Keyword MATCH contributes its BM25 score to
score() instead of 0.
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
