kuramitsu commented on code in PR #12885:
URL: https://github.com/apache/lucene/pull/12885#discussion_r1421180794
##########
lucene/analysis/kuromoji/src/java/org/apache/lucene/analysis/ja/JapaneseReadingFormFilter.java:
##########
@@ -43,10 +43,30 @@ public JapaneseReadingFormFilter(TokenStream input) {
this(input, false);
}
+ private boolean isAllHiragana(CharSequence s) {
+ int len = s.length();
+ for (int i = 0; i < len; i++) {
+ char ch = s.charAt(i);
+ if (ch < 0x3040 || ch > 0x3097) {
+ return false;
+ }
+ }
+ return true;
+ }
+
@Override
public boolean incrementToken() throws IOException {
if (input.incrementToken()) {
String reading = readingAttr.getReading();
+ if (reading == null & isAllHiragana(termAttr)) {
Review Comment:
Fixed to support partial hiragana.
https://github.com/apache/lucene/pull/12885/commits/2967e388c4f773a76268c8e8001a1dae1fec62c6
However, it is difficult to describe a test for OOV terms that contain a
mixture of hiragana and katakana, because the system seems to basically split
different character types for OOV terms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
