Got it. Will do so, and amend my JIRA ticket to include this as well as tests.
Thanks! On Sat, May 17, 2014 at 2:21 AM, Uwe Schindler <[email protected]> wrote: > Hi, > > > > you have to capture state on the first token before inserting new ones. > When inserting a new token, **solely** call restoreState(); > clearAttributes() is not needed before restoreState(). > > If you don’t do this, your filter will work incorrect if other filters > come **after** it. > > > > The assertion in BaseTokenStreamTestCase is therefore correct and really > mandatory. There are many filters that show how to do this token inserting > correctly. > > > > Uwe > > > > ----- > > Uwe Schindler > > H.-H.-Meier-Allee 63, D-28213 Bremen > > http://www.thetaphi.de > > eMail: [email protected] > > > > *From:* Nitzan Shaked [mailto:[email protected]] > *Sent:* Friday, May 16, 2014 6:28 AM > *To:* [email protected] > *Subject:* BaseTokenStreamTestCase > > > > Hi all > > > > While writing the unit tests for a new token filter I came across an > issue(?) with BaseTokenStreamTestCase.assertTokenStreamContents(): it goes > to some length to assure that clearAttributes() was called for every token > produced by the filter under test. > > > > I suppose this helps most of the time, but my filter produces sometimes > more than 1 output token for a given input token. I don't want to care > about what attributes the input token carries, and so don't clear > attributes between producing the output tokens from a given input token: I > only change the attributes I care about (in my case this is charTerm right > now, and nothing else, not even positionIncrement). > > > > This makes my unit tests unable to use > BaseTokenStreamTestCase.assertTokenStreamContents(). I certainly do not > want to add a captureState() and "clearAttributes() ; restoreState() " > calls just so I can pass the unit tests. > > > > I would rather change assertTokenStreamContents to support my use case, by > adding a boolean and making the required changes everywhere else. > > > > Thoughts? > > Nitzan > > >
