David Smiley created LUCENE-6874:
------------------------------------
Summary: WhitespaceTokenizer should tokenize on NBSP
Key: LUCENE-6874
URL: https://issues.apache.org/jira/browse/LUCENE-6874
Project: Lucene - Core
Issue Type: Improvement
Components: modules/analysis
Reporter: David Smiley
Priority: Minor
WhitespaceTokenizer uses [Character.isWhitespace
|http://docs.oracle.com/javase/8/docs/api/java/lang/Character.html#isWhitespace-int-]
to decide what is whitespace. Here's a pertinent excerpt:
bq. It is a Unicode space character (SPACE_SEPARATOR, LINE_SEPARATOR, or
PARAGRAPH_SEPARATOR) but is not also a non-breaking space ('\u00A0', '\u2007',
'\u202F')
Perhaps Character.isWhitespace should have been called
isLineBreakableWhitespace?
I think WhitespaceTokenizer should tokenize on this. I am aware it's easy to
work around but why leave this trap in by default?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
