github-actions[bot] commented on code in PR #69698:
URL: https://github.com/apache/doris/pull/69698#discussion_r4396191559
##########
be/src/storage/index/inverted/reader/collection_statistics.cpp:
##########
@@ -117,6 +119,65 @@ void add_term_doc_frequency(
(*logical_frequencies)[field][logical_term] += doc_frequency;
}
+Status segment_scoring_terms(index_query::IndexSource& source, const
CollectInfo& collect_info,
+ const VisibleDocFreq& visible_doc_freq,
+ FieldPrefixExpansions* expansions,
std::vector<std::string>* terms) {
+ DORIS_CHECK(expansions != nullptr);
+ DORIS_CHECK(terms != nullptr);
+ *terms = collect_info.unique_terms;
+ std::unordered_set<std::string> listed(terms->begin(), terms->end());
+ for (const auto& scoring_prefix : collect_info.scoring_prefixes) {
+ index_query::TermPattern pattern;
+
RETURN_IF_ERROR(index_query::TermPattern::create(index_query::TermPatternKind::kPrefix,
+
scoring_prefix.prefix, &pattern));
+ const auto wanted = static_cast<size_t>(scoring_prefix.max_expansions);
+ std::vector<std::string> visible;
+ for (int32_t limit = scoring_prefix.max_expansions;; limit *= 2) {
+ std::vector<std::string> expanded;
+ RETURN_IF_ERROR(source.expand_terms(pattern, limit, &expanded));
+ visible.clear();
+ for (auto& term : expanded) {
+ // The statistics are keyed by wide string, and
StringHelper::to_wstring drops the
+ // bytes that are not valid UTF-8, which SNII keeps in a
token. Two such terms,
+ // "ra\xFF" and "ra\xFE", would share the key "ra" and add
their document
+ // frequencies up, to more than the documents of the tablet
and a negative idf. A
+ // term without a key of its own is not scored, and takes no
place.
+ if (segment_v2::inverted_index::StringHelper::to_string(
+
segment_v2::inverted_index::StringHelper::to_wstring(term)) != term) {
+ continue;
+ }
+ uint64_t doc_freq = 0;
+ RETURN_IF_ERROR(visible_doc_freq(term, &doc_freq));
Review Comment:
[P2] Reuse visible frequencies instead of intersecting postings twice
On a merge-on-write segment with hidden rows, `visible_doc_freq` calls
`exclude_hidden_term_docs` / `scoring_doc_freq_among`, opening a fresh postings
cursor and counting its intersection with the hidden rows. Every retained
expansion is checked here, then checked again in the SNII and CLucene
accumulation loops (lines 595 and 739), discarding the count already computed.
With the default cap of 50 common expansions and dense hidden rows, this
traverses those expansion postings twice per segment before scoring starts; the
dictionary caches do not cache hidden-document counts. Refill rounds and
overlapping prefixes repeat the same work further. Cache the visible frequency
per raw term for the segment/field, or return it from this helper, so selection
and final accumulation reuse one intersection result in both formats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
