airborne12 opened a new pull request, #69815:
URL: https://github.com/apache/doris/pull/69815
### What problem does this PR solve?
Issue Number: None
Related PR: #69052
Problem Summary: score() of pattern queries disagrees with Elasticsearch
and with itself:
-- 'Inverted indexes are suitable for full text search',
-- 'The parser splits text into multiple terms', ...
SELECT score() FROM document WHERE SEARCH('content:/.*ext/'); -- BM25
sum
SELECT score() FROM document WHERE SEARCH('content:*ext'); -- BM25
sum
SELECT score() FROM document WHERE SEARCH('content:te*'); -- 1
SELECT score() FROM document WHERE content MATCH_REGEXP 'ext'; -- 0
Elasticsearch scores a query_string /regexp/, wildcard or prefix, and a
regexp, wildcard or prefix query, with a constant 1 per matching row
(default rewrite constant_score_blended); measured on 9.5.5 with the
rows above: 1.0 for both matching rows. Only an explicit
"rewrite": "scoring_boolean" sums the BM25 of the expanded terms. Next
to a scored clause in a bool must, a regexp adds exactly 1.
#69052 made SEARCH regexp and wildcard sum the BM25 of every expanded
term, i.e. scoring_boolean always on, while prefix stayed constant. That
sum also takes idf and avgdl from the current segment, not from the
tablet statistics MATCH uses: two inserts of the same text scored
0.2876821 and 1.3862942 for identical 'apple' rows, while MATCH 'apple'
scored 0.6931471 for both. MATCH_REGEXP never requested a scoring
context, so it scored 0 like a filter.
Fix:
1. Revert the scoring part of #69052: ExpandWeight scores every pattern
with the constant scorer again. The test fixture extensions that
#69051 built on are kept.
2. MATCH_REGEXP requests scoring on every index; the regexp expansion
then publishes the constant 1. The MATCH statistics collector skips
it, since a constant reads no statistics and the pattern is not text
to analyze.
### Release note
score() of SEARCH regexp and wildcard clauses is a constant 1 per
matching row again, like prefix clauses and Elasticsearch's default.
score() of MATCH_REGEXP is 1 instead of 0.
### Check List (For Author)
- Test: Regression test / Unit Test / Manual test
- Regression test: new search/test_search_score_pattern_constant
(V2/V3/SNII, two segments, keyword and analyzed fields; identical
on all three formats); 24 existing score suites in
inverted_index_p0, search and variant_p0 pass unchanged
- Unit Test: 477 tests (IndexReaderHelperTest, CollectionStatistics*,
regexp/wildcard/prefix engine tests restored by the revert,
FunctionSearchTest, SNII query tests). Mutation checks: the gate
test fails without the gate change, the collector test fails
without the regexp skip
- Manual test: Elasticsearch 9.5.5 on the same rows for every query
shape above; two-segment table now scores 1 for every pattern hit
- Built and run on master 62e3d81a42 plus this change: the current
master tip does not compile (function_datasketches_hll.cpp after
#69679), and none of the files touched here changed in between
- Behavior changed: Yes. SEARCH regexp/wildcard scores and MATCH_REGEXP
scores change as described in the release note.
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
