Hello,
 
I'm using Lucene for a few weeks now in a small project and just ran
into a problem. My index contains words that contain one or more
underlines, e.g. XYZZZY_DE_SA0001 or XYZZZY_AT0001. Unfortunately the
tokenizer tokenizes / splits the word into multiple tokens at the
underscores, except the last underscore. 
 
For example the word XYZZZY_DE_SA0001 is tokenized as follows:
 
1. Token: XYZZY 
2. Token: DE_SA0001
 
which is not conforming to expectations. Either the tokenizer should
split at every underscore or at none.
 
I'm using Lucene 1.4.3 with
org.apache.lucene.analysis.standard.StandardAnalyzer and Java 1.4.2_08.
 
Has anybody experienced the same behaviour or can explain it? Could it
be a bug in the StandardTokenizer?
 
Many thanks in advance
 
Sebastian Seitz

Reply via email to