github-actions[bot] commented on code in PR #69790:
URL: https://github.com/apache/doris/pull/69790#discussion_r4413114719
##########
be/src/storage/index/inverted/reader/predicate_collector.cpp:
##########
@@ -285,6 +286,32 @@ void preserve_selected_index_metadata(const
ScoringIndexCandidates& candidates,
Review Comment:
[P1] Do not score untyped VARIANT paths as text
When an extracted VARIANT path has no materialized type,
`resolve_text_scoring_index_candidates()` forces it to STRING and clones the
parent index. A dynamic numeric path can inherit metadata with parser/analyzer
removed and be written as BKD, while `CAST(v['n'] AS STRING) MATCH '1'` is
excluded from index execution because the source type is VARIANT. This gate
nevertheless admits the statistics collector, which can open that BKD file
through the CLucene/SNII text-statistics path and fail before row-level
filtering. Keep unproven VARIANT paths out of scoring or prove the physical
reader type, and add a dynamic numeric VARIANT cast regression.
##########
be/src/storage/index/inverted/reader/predicate_collector.cpp:
##########
@@ -285,6 +286,32 @@ void preserve_selected_index_metadata(const
ScoringIndexCandidates& candidates,
using namespace scoring_collection_detail;
+namespace {
+
+// The terms a MATCH scores, the ones its reader looks up. A keyword index
stores the whole value
+// as one term, case and spaces included: tag MATCH 'Hello World' scores the
single term
+// "Hello World", as SEARCH('tag:"Hello World"') does. Any other index cuts
the value with the
+// query's analyzer.
+Result<std::vector<TermInfo>> match_scoring_terms(const std::string& value,
+ InvertedIndexQueryType
query_type,
+ const TabletIndex&
index_meta,
+ const
InvertedIndexAnalyzerCtx& analyzer_ctx) {
+ if
(!inverted_index::InvertedIndexAnalyzer::should_analyzer(index_meta.properties()))
{
Review Comment:
[P2] Preserve raw-byte identity for keyword MATCH statistics
The keyword branch now sends the literal bytes to collection statistics, but
direct terms are converted through `StringHelper::to_wstring()` without the
round-trip check used for prefix expansions. That helper drops invalid UTF-8
bytes (the existing test documents `ra\xFF` and `ra\xFE` collapsing to `ra`),
while SNII keeps the original bytes in its dictionary. Distinct malformed
keyword values can therefore share one statistics key and give `score()` the
wrong IDF, even though the reader looks up the raw term. Reject or preserve
non-round-tripping direct terms and add a keyword MATCH score regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
