I want to implement an analyzer which use WhitespaceAnalyzer first
then my tokenFilter. But my filter need not global information of
token such as how many times a token occur. So in tokenStream method.
I iterate the tokenStream to get all the things I need. Then
pass this information to my own TokenFilter. But my tokenfilter's
incrementToken method will always get nothing from the TokenStream
because it's already consumed.
public MyAnalyzer{
...
Analyzer wa=new WhitespaceAnalyzer();
public TokenStream tokenStream(String fieldName, Reader reader) {
TokenStream tokenStream=null;
try {
tokenStream = wa.reusableTokenStream(fieldName, reader);
while(tokenStream.incrementToken()){
//do something
}
} catch (IOException e1) {
e1.printStackTrace();
}
/*
try {
tokenStream.reset();
//tokenStream =
wa.reusableTokenStream(fieldName, reader);
while(tokenStream.incrementToken()){
//it don't come here, which means the
tokenStream is not resuable.
System.out.println("resuable");
}
} catch (IOException e) {
e.printStackTrace();
}
*/
MyFilter stream = new MyFilter(tokenStream);
return stream;
}
}
2010/5/28 Erick Erickson <erickerick...@gmail.com>:
> What is the problem you're seeing? Maybe a stack trace?
> You haven't told us what the incorrect behavior is.
>
> Best
> Erick
>
> On Fri, May 28, 2010 at 12:52 AM, Li Li <fancye...@gmail.com> wrote:
>
>> I want to analyzer a text twice so that I can get some statistic
>> information from this text
>> TokenStream tokenStream=null;
>> Analyzer wa=new WhitespaceAnalyzer();
>> try {
>> tokenStream = wa.reusableTokenStream(fieldName,
>> reader);
>> while(tokenStream.incrementToken()){
>> //do something
>> }
>> } catch (IOException e1) {
>> e1.printStackTrace();
>> }
>>
>> try {
>> tokenStream.reset();
>> //tokenStream = wa.reusableTokenStream(fieldName,
>> reader);
>> while(tokenStream.incrementToken()){
>> //not here
>> System.out.println("resuable");
>>
>> }
>> } catch (IOException e) {
>> // TODO Auto-generated catch block
>> e.printStackTrace();
>> }
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
