airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4065357596
##########
be/src/storage/index/inverted/tokenizer/icu/icu_tokenizer.cpp:
##########
@@ -55,13 +55,23 @@ Token* ICUTokenizer::next(Token* token) {
utf8Str_.clear();
int32_t length = std::min(end - start, LUCENE_MAX_WORD_LEN);
auto subString = buffer_.tempSubString(start, length);
+ sourceUtf8Str_.clear();
+ subString.toUTF8String(sourceUtf8Str_);
if (this->lowercase) {
subString.toLower().toUTF8String(utf8Str_);
} else {
subString.toUTF8String(utf8Str_);
}
token->setNoCopy(utf8Str_.data(), 0,
static_cast<int32_t>(utf8Str_.size()));
+ int32_t source_start = 0;
+ int32_t source_end = 0;
+ if (start >= 0 && length >= 0 && advance_source_offset(start,
source_start) &&
Review Comment:
Fixed in 31450230. ICU reset now validates raw UTF-8 before ICU conversion
and throws `INVALID_ARGUMENT` for malformed input, preventing replacement text
and stale token spans. The test covers an invalid byte between valid words and
successful reuse after a valid reset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
