github-actions[bot] commented on code in PR #69908:
URL: https://github.com/apache/doris/pull/69908#discussion_r4430289029
##########
be/src/storage/index/inverted/snii/bkd/bkd_reader.cpp:
##########
@@ -304,14 +412,25 @@ Status BkdReader::range(Slice lower, bool
lower_inclusive, Slice upper, bool upp
// bound on its own side.
RETURN_IF_ERROR(
scan_boundary_leaf(first, lower, lower_inclusive, Slice(), true,
hits, scratch));
- // Gather doc IDs from fully covered leaves, sort them, and insert them in
bounded chunks. Leaf order follows values rather than doc IDs, and bounded
chunks limit memory for large ranges.
+ RETURN_IF_ERROR(add_whole_leaves(first + 1, last, hits, scratch));
+ return scan_boundary_leaf(last, Slice(), true, upper, upper_inclusive,
hits, scratch);
+}
+
+Status BkdReader::add_whole_leaves(uint32_t first, uint32_t end,
roaring::Roaring* hits,
+ BkdQueryScratch* scratch) const {
+ // Leaf order follows values rather than doc IDs. A narrow range gathers
doc IDs, sorts them and
+ // inserts them in bounded chunks, which limit memory.
constexpr size_t kMaxGatheredDocIds = 1U << 20;
+ const bool use_doc_bits =
+ static_cast<uint64_t>(end - first) *
block_.header().points_per_leaf >=
Review Comment:
[P2] Avoid selecting the flat bitmap solely from truncated `doc_count /
256`. For a valid index with fewer than 256 distinct docs, this expression is
zero, so any whole-leaf range takes the bitmap path. If one row id is `(1U <<
28) - 1` and its array values span three leaves, `set_doc_bits` grows
`doc_bits` to 4,194,304 words (32 MiB) for that single hit, and reusable
scratch keeps that capacity after `clear()`. Gate the bitmap on the actual
point/doc-id density or maximum id (and account the temporary image) so sparse
high ids use the existing gathered sort path without this per-query memory
spike.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
