github-actions[bot] commented on code in PR #70263:
URL: https://github.com/apache/doris/pull/70263#discussion_r4581552136


##########
be/src/storage/index/inverted/analysis/token_filter/word_delimiter_filter.cpp:
##########
@@ -190,75 +168,31 @@ Token* WordDelimiterFilter::next(Token* t) {
     }
 }
 
-void WordDelimiterFilter::reset() {
-    DorisTokenFilter::reset();
+void WordDelimiterFilter::on_reset() {
     _has_saved_state = false;
-    _current_generated = false;
     _concat->clear();
     _concat_all->clear();
     _accum_pos_inc = 0;
     _buffered_pos = 0;
     _buffered_len = 0;
     _first = true;
-    _saved_source_byte_offsets.clear();
-    _saved_source_byte_end_offsets.clear();
-    _saved_token_byte_offsets.clear();
-    _current_source_byte_offsets.clear();
-    _current_source_byte_end_offsets.clear();
-    release_oversized_scratch(_saved_source_byte_offsets);
-    release_oversized_scratch(_saved_source_byte_end_offsets);
-    release_oversized_scratch(_saved_token_byte_offsets);
-    release_oversized_scratch(_current_source_byte_offsets);
-    release_oversized_scratch(_current_source_byte_end_offsets);
     _concat->release_oversized_buffers();
     _concat_all->release_oversized_buffers();
-    auto release_attribute = [](Attribute& attribute) {
-        release_oversized_scratch(attribute.buffered);
-        release_oversized_scratch(attribute.source_byte_offsets);
-        release_oversized_scratch(attribute.source_byte_end_offsets);
-    };
-    release_attribute(_attribute);
+    release_oversized_scratch(_attribute.buffered);
     if (_states.capacity() * sizeof(Attribute) > 
ANALYZER_SCRATCH_HIGH_WATER_BYTES) {
         std::vector<Attribute>(INITIAL_BUFFERED_STATES).swap(_states);
     } else {
-        std::ranges::for_each(_states, release_attribute);
+        std::ranges::for_each(_states,

Review Comment:
   [P2] Bound the combined capacity of buffered subword strings
   
   With the supported `empty` tokenizer followed by default `word_delimiter`, a 
value containing 512 4 KiB ASCII words separated by `-` leaves at least 2 MiB 
in `_states[*].buffered`. The `Attribute` array is only about 20 KiB and each 
string is below 64 KiB, so resetting to a short value releases neither; the 
reused writer stream keeps those buffers until the segment ends. This becomes a 
regression here because `LuceneBackedTextAnalyzer` previously destroyed these 
components per value, whereas `ChainAnalyzer` now reuses them. Include the 
nested string capacities in the reset budget, and cover this 
moderate-count/large-subword case in the retention test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to