github-actions[bot] commented on code in PR #69698:
URL: https://github.com/apache/doris/pull/69698#discussion_r4396102407
##########
be/src/storage/index/inverted/reader/collection_statistics.cpp:
##########
@@ -117,6 +119,56 @@ void add_term_doc_frequency(
(*logical_frequencies)[field][logical_term] += doc_frequency;
}
+Status segment_scoring_terms(index_query::IndexSource& source, const
CollectInfo& collect_info,
+ const VisibleDocFreq& visible_doc_freq,
+ FieldPrefixExpansions* expansions,
std::vector<std::string>* terms) {
+ DORIS_CHECK(expansions != nullptr);
+ DORIS_CHECK(terms != nullptr);
+ *terms = collect_info.unique_terms;
+ std::unordered_set<std::string> listed(terms->begin(), terms->end());
+ for (const auto& scoring_prefix : collect_info.scoring_prefixes) {
+ index_query::TermPattern pattern;
+
RETURN_IF_ERROR(index_query::TermPattern::create(index_query::TermPatternKind::kPrefix,
+
scoring_prefix.prefix, &pattern));
+ const auto wanted = static_cast<size_t>(scoring_prefix.max_expansions);
+ std::vector<std::string> visible;
+ for (int32_t limit = scoring_prefix.max_expansions;; limit *= 2) {
+ std::vector<std::string> expanded;
+ RETURN_IF_ERROR(source.expand_terms(pattern, limit, &expanded));
+ visible.clear();
+ for (auto& term : expanded) {
+ uint64_t doc_freq = 0;
+ RETURN_IF_ERROR(visible_doc_freq(term, &doc_freq));
+ if (doc_freq > 0) {
+ visible.push_back(std::move(term));
+ }
+ }
+ // The dictionary ran out of terms, or the first `wanted` visible
ones are found.
+ if (expanded.size() < static_cast<size_t>(limit) || visible.size()
>= wanted) {
+ break;
+ }
+ }
+ // A read with doubled limit can find more visible terms than the
prefix keeps; the
+ // dictionary order puts the ones to keep first.
+ if (visible.size() > wanted) {
+ visible.resize(wanted);
+ }
+ auto& prefix_terms =
+ expansions
+
->try_emplace(segment_v2::inverted_index::StringHelper::to_wstring(
+ scoring_prefix.prefix),
+ scoring_prefix.max_expansions)
+ .first->second;
+ for (auto& term : visible) {
+
prefix_terms.insert(segment_v2::inverted_index::StringHelper::to_wstring(term));
Review Comment:
[P2] Preserve raw SNII term identity in prefix statistics
SNII preserves raw token bytes, but this conversion uses Boost's default
[skip-invalid
policy](https://www.boost.org/doc/libs/1_84_0/libs/locale/doc/html/group__codepage.html),
so distinct dictionary terms can become the same key. A whitespace
`char_group` analyzer with no filters and `support_phrase=true` can index one
row containing `CONCAT('apple ra r', UNHEX('FF'), 'a r', UNHEX('FE'), 'a')`:
the tokenizer retains invalid bytes inside each token. For the valid query
`MATCH_PHRASE_PREFIX 'apple r'` at the default cap, all three distinct `r...a`
expansions become `L"ra"`. The frequency loop counts each raw term once and
accumulates them under that wide key, yielding `df(ra)=3` with `N=1`; the
per-raw-term SNII bounds checks do not catch this collision. The scorer then
uses `idf(apple)+idf(ra)=log(4/3)+log(4/7)=-0.2719`, so `score()>0` drops the
matching row. Before this change the ASCII query did not collect those
malformed dictionary terms. Preserve lossless raw-term identity an
d dictionary order through expansion storage and frequency lookup, and add
SNII coverage for malformed dictionary tokens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
