github-actions[bot] commented on code in PR #70263:
URL: https://github.com/apache/doris/pull/70263#discussion_r4581552136
##########
be/src/storage/index/inverted/analysis/token_filter/word_delimiter_filter.cpp:
##########
@@ -190,75 +168,31 @@ Token* WordDelimiterFilter::next(Token* t) {
}
}
-void WordDelimiterFilter::reset() {
- DorisTokenFilter::reset();
+void WordDelimiterFilter::on_reset() {
_has_saved_state = false;
- _current_generated = false;
_concat->clear();
_concat_all->clear();
_accum_pos_inc = 0;
_buffered_pos = 0;
_buffered_len = 0;
_first = true;
- _saved_source_byte_offsets.clear();
- _saved_source_byte_end_offsets.clear();
- _saved_token_byte_offsets.clear();
- _current_source_byte_offsets.clear();
- _current_source_byte_end_offsets.clear();
- release_oversized_scratch(_saved_source_byte_offsets);
- release_oversized_scratch(_saved_source_byte_end_offsets);
- release_oversized_scratch(_saved_token_byte_offsets);
- release_oversized_scratch(_current_source_byte_offsets);
- release_oversized_scratch(_current_source_byte_end_offsets);
_concat->release_oversized_buffers();
_concat_all->release_oversized_buffers();
- auto release_attribute = [](Attribute& attribute) {
- release_oversized_scratch(attribute.buffered);
- release_oversized_scratch(attribute.source_byte_offsets);
- release_oversized_scratch(attribute.source_byte_end_offsets);
- };
- release_attribute(_attribute);
+ release_oversized_scratch(_attribute.buffered);
if (_states.capacity() * sizeof(Attribute) >
ANALYZER_SCRATCH_HIGH_WATER_BYTES) {
std::vector<Attribute>(INITIAL_BUFFERED_STATES).swap(_states);
} else {
- std::ranges::for_each(_states, release_attribute);
+ std::ranges::for_each(_states,
Review Comment:
[P2] Bound the combined capacity of buffered subword strings
With the supported `empty` tokenizer followed by default `word_delimiter`, a
value containing 512 4 KiB ASCII words separated by `-` leaves at least 2 MiB
in `_states[*].buffered`. The `Attribute` array is only about 20 KiB and each
string is below 64 KiB, so resetting to a short value releases neither; the
reused writer stream keeps those buffers until the segment ends. This becomes a
regression here because `LuceneBackedTextAnalyzer` previously destroyed these
components per value, whereas `ChainAnalyzer` now reuses them. Include the
nested string capacities in the reset budget, and cover this
moderate-count/large-subword case in the retention test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]