github-actions[bot] commented on code in PR #69790:
URL: https://github.com/apache/doris/pull/69790#discussion_r4408148364
##########
be/src/storage/index/inverted/reader/predicate_collector.cpp:
##########
@@ -285,6 +285,24 @@ void preserve_selected_index_metadata(const
ScoringIndexCandidates& candidates,
using namespace scoring_collection_detail;
+namespace {
+
+// The terms a MATCH scores, the ones its reader looks up. A keyword index
stores the whole value
+// as one term, case and spaces included: tag MATCH 'Hello World' scores the
single term
+// "Hello World", as SEARCH('tag:"Hello World"') does. Any other index cuts
the value with the
+// query's analyzer.
+Result<std::vector<TermInfo>> match_scoring_terms(const std::string& value,
+ const TabletIndex&
index_meta,
+ const
InvertedIndexAnalyzerCtx& analyzer_ctx) {
+ if
(!inverted_index::InvertedIndexAnalyzer::should_analyzer(index_meta.properties()))
{
+ return std::vector<TermInfo> {TermInfo(value)};
Review Comment:
[P1] Strip MATCH_PHRASE slop before collecting keyword score terms
`MATCH_PHRASE` supports a trailing `~N`/`~N+`, and `lower_match()` removes
that suffix before the keyword reader looks up the term (for example, `tag
MATCH_PHRA 'apple ~1'` executes the term `apple`). This new keyword branch
records the entire raw literal as `apple ~1`, so collection statistics use an
absent/different term and compute the wrong IDF for every matching row. Reuse
the same slop parsing as the reader before creating the `TermInfo`, and add a
`score()` regression for keyword phrase slop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
