Re: Question about the CompoundWordTokenFilterBase

2013-09-18 Thread Jack Krupansky
escu Sent: Wednesday, September 18, 2013 10:27 AM To: java-user@lucene.apache.org Subject: Question about the CompoundWordTokenFilterBase Hi, While trying to play with the CompoundWordTokenFilterBase I noticed that the behavior is to include the original token together with the new sub-tokens. I assu

Question about the CompoundWordTokenFilterBase

2013-09-18 Thread Alex Parvulescu
Hi, While trying to play with the CompoundWordTokenFilterBase I noticed that the behavior is to include the original token together with the new sub-tokens. I assume this is expected (haven't found any relevant docs on this), but I was wondering if it's a hard requirement or can I propose a small