Ryan19929 commented on code in PR #50141:
URL: https://github.com/apache/doris/pull/50141#discussion_r2063641659
##########
be/src/olap/rowset/segment_v2/inverted_index/analyzer/ik/core/CharacterUtil.cpp:
##########
@@ -20,77 +20,50 @@
namespace doris::segment_v2 {
int32_t CharacterUtil::identifyCharType(int32_t rune) {
- // Numbers
- if (rune >= 0x30 && rune <= 0x39) {
+ if (rune >= '0' && rune <= '9') {
return CHAR_ARABIC;
}
-
- // English
- if ((rune >= 0x61 && rune <= 0x7a) || (rune >= 0x41 && rune <= 0x5a)) {
+ if ((rune >= 'a' && rune <= 'z') || (rune >= 'A' && rune <= 'Z')) {
return CHAR_ENGLISH;
}
- // CJK Unified Chinese Characters
- if ((rune >= 0x4E00 && rune <= 0x9FFF) || (rune >= 0x3400 && rune <=
0x4DBF) ||
- (rune >= 0x20000 && rune <= 0x2A6DF) || (rune >= 0x2A700 && rune <=
0x2B73F) ||
- (rune >= 0x2B740 && rune <= 0x2B81F) || (rune >= 0x2B820 && rune <=
0x2CEAF) ||
- (rune >= 0x2CEB0 && rune <= 0x2EBEF) || (rune >= 0x30000 && rune <=
0x3134F)) {
+ UBlockCode block = ublock_getCode(rune);
+
+ if (block == UBLOCK_CJK_UNIFIED_IDEOGRAPHS || block ==
UBLOCK_CJK_COMPATIBILITY_IDEOGRAPHS ||
Review Comment:
After expanding the check scope (e.g., to include Blocks B–G), extended CJK
characters (like "𠮷") that were segmented as standalone tokens in the Java
version might be merged with adjacent text in the C++ implementation. For
example, "A𠮷B" would originally split into `[A]`,`[𠮷]`,`[B]` but could become
`[A]`,`[𠮷B]`. Is this behavior acceptable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
