The indexer only call getAttribute/addAttribute one time after initializing (see docs). It will never call it later. If you cache tokens, you always have to restore the state into the TokenStream's attributes.
----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Daniel Shane [mailto:sha...@lexum.umontreal.ca] > Sent: Thursday, September 03, 2009 8:55 PM > To: java-user@lucene.apache.org > Subject: TokenStream API, Quick Question. > > Does a TokenStream have to return always the same number of attributes > with the same underlying classes for all the tokens it generates? > > I mean, during the tokenization phase, can the first "token" have a Term > and Offset Attribute and the second "token" only a Type Attribute or > does this mean that the first token has to have an empty Type attribute > as well? > > I'm just not sure, > Daniel Shane > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
