airborne12 commented on code in PR #69698:
URL: https://github.com/apache/doris/pull/69698#discussion_r4396031443
##########
be/src/storage/index/inverted/reader/collection_statistics.cpp:
##########
@@ -801,6 +854,27 @@ float CollectionStatistics::get_or_calculate_avg_dl(const
std::wstring& lucene_c
return avg_dl;
}
+std::vector<std::wstring> CollectionStatistics::get_prefix_expansions(
+ const std::wstring& lucene_col_name, const std::wstring& prefix) {
+ const auto field = _prefix_expansions.find(lucene_col_name);
+ if (field == _prefix_expansions.end()) {
+ return {};
+ }
+ const auto expansions = field->second.find(prefix);
+ if (expansions == field->second.end()) {
+ return {};
+ }
+ // A merge-on-write scan hides rows: a term whose documents are all hidden
was expanded in
+ // its segment but has no document frequency left, and its idf would be
that of an absent term.
+ std::vector<std::wstring> terms;
+ for (const auto& term : expansions->second.terms()) {
+ if (get_term_doc_freq_by_col(lucene_col_name, term) > 0) {
Review Comment:
Fixed in 6edc0096a7. The cap is now applied to visible terms: the segment
asks for each expanded term's document frequency without its hidden rows and
keeps the first `max_expansions` terms that have a visible document. While the
dictionary has more terms and fewer visible ones were found, it reads again
with twice the limit, so the extra read happens only when hidden-only terms are
met. The late filter in `get_prefix_expansions` is gone, since every kept term
already has a visible document. A kept term is still among the smallest visible
terms of every segment holding it, so the earlier frequency argument holds.
Your example, checked on a merge-on-write table (limit 1; row 1 updated so
that `apple ra` is hidden; `apple rb` visible): `apple r` scores the rows at
1.0498 both before and after a full compaction (5 physical rows before, 4
after).
Unit coverage in `phrase_prefix_query_test`: a term with no visible document
takes no place (limit 1 expands to the next term), and a prefix whose terms are
all hidden expands to nothing. I did not add a before/after-compaction
regression case; the check above was manual.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
