Hi, While trying to play with the CompoundWordTokenFilterBase I noticed that the behavior is to include the original token together with the new sub-tokens.
I assume this is expected (haven't found any relevant docs on this), but I was wondering if it's a hard requirement or can I propose a small change to skip the original token (controlled by a flag)? If there's interest I can put this in a JIRA issue and we can continue the discussion there. The patch is not too complicated, but I haven't ran any of the tests yet :) thanks, alex
