Nitzan Shaked created LUCENE-5674:
-------------------------------------
Summary: A new token filter: SubSequence
Key: LUCENE-5674
URL: https://issues.apache.org/jira/browse/LUCENE-5674
Project: Lucene - Core
Issue Type: Improvement
Components: core/other
Reporter: Nitzan Shaked
Priority: Minor
Attachments: 0001-SubSeqFilter.patch
A new configurable token filter which, given a token breaks it into sub-parts
and outputs consecutive sub-sequences of those sub-parts.
Useful for, for example, using during indexing to generate variations on domain
names, so that "www.google.com" can be found by searching for "google.com", or
"www.google.com".
Parameters:
sepRegexp: A regular expression used split incoming tokens into sub-parts.
glue: A string used to concatenate sub-parts together when creating
sub-sequences.
minLen: Minimum length (in sub-parts) of output sub-sequences
maxLen: Maximum length (in sub-parts) of output sub-sequences (0 for unlimited;
negative numbers for token length in sub-parts minus specified length)
anchor: Anchor.START to output only prefixes, or Anchor.END to output only
suffixes, or Anchor.NONE to output any sub-sequence
withOriginal: whether to output also the original token
Note: tests will follow, I currently believe BaseTokenStreamTestCase is
broken/mis-designed for such use cases, I will open a separate Jira for that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
