airborne12 commented on code in PR #69698:
URL: https://github.com/apache/doris/pull/69698#discussion_r4396258286
##########
be/src/storage/index/inverted/reader/collection_statistics.cpp:
##########
@@ -117,6 +119,65 @@ void add_term_doc_frequency(
(*logical_frequencies)[field][logical_term] += doc_frequency;
}
+Status segment_scoring_terms(index_query::IndexSource& source, const
CollectInfo& collect_info,
+ const VisibleDocFreq& visible_doc_freq,
+ FieldPrefixExpansions* expansions,
std::vector<std::string>* terms) {
+ DORIS_CHECK(expansions != nullptr);
+ DORIS_CHECK(terms != nullptr);
+ *terms = collect_info.unique_terms;
+ std::unordered_set<std::string> listed(terms->begin(), terms->end());
+ for (const auto& scoring_prefix : collect_info.scoring_prefixes) {
+ index_query::TermPattern pattern;
+
RETURN_IF_ERROR(index_query::TermPattern::create(index_query::TermPatternKind::kPrefix,
+
scoring_prefix.prefix, &pattern));
+ const auto wanted = static_cast<size_t>(scoring_prefix.max_expansions);
+ std::vector<std::string> visible;
+ for (int32_t limit = scoring_prefix.max_expansions;; limit *= 2) {
+ std::vector<std::string> expanded;
+ RETURN_IF_ERROR(source.expand_terms(pattern, limit, &expanded));
+ visible.clear();
+ for (auto& term : expanded) {
+ // The statistics are keyed by wide string, and
StringHelper::to_wstring drops the
+ // bytes that are not valid UTF-8, which SNII keeps in a
token. Two such terms,
+ // "ra\xFF" and "ra\xFE", would share the key "ra" and add
their document
+ // frequencies up, to more than the documents of the tablet
and a negative idf. A
+ // term without a key of its own is not scored, and takes no
place.
+ if (segment_v2::inverted_index::StringHelper::to_string(
+
segment_v2::inverted_index::StringHelper::to_wstring(term)) != term) {
+ continue;
+ }
+ uint64_t doc_freq = 0;
+ RETURN_IF_ERROR(visible_doc_freq(term, &doc_freq));
Review Comment:
Fixed in the commit that follows 59f3dc7db3.
`collection_statistics_detail::cached_visible_doc_freq` computes the frequency
once per raw term for the segment and the field, and returns it to both users:
the choice of the expansions (`segment_scoring_terms`) and the accumulation
loop, in CLucene and in SNII. The cache lives as long as the returned function,
so refill rounds and a second prefix of the same field reuse it too.
`phrase_prefix_query_test` counts the source's `doc_freq` calls: the terms
of a prefix are asked for in the choice and again in the count, and each term
is read once. The merge-on-write check is unchanged by this: the table with a
hidden `apple ra` still scores the same rows at 1.0498 before and after a full
compaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
