I see some definitions in StandardTokenizerConstants.java
Perhaps these are the values for t.kind.
Perhaps, I was confused between between the usage of
getNextToken() and next() in the standard analyzer.
When should one use getNextToken() instead of next()
I am just starting to use Lucene, so please excuse these
simple questions.
Thanks
On 9/21/05, Beady Geraghty <[EMAIL PROTECTED]> wrote:
>
> Could someone tell me how to use the StandardTokenizer properly ?
> I thought that if the tokenizer.getNextToken() returns null, then it is
> the end of stream. I have a loop that tries to get the next token until
> it is null. But the loop doesn't terminate.
> I tried to termintae the loop by t.kind == 0, and it seems to have stopped
> upon the end of stream. I am not sure what t.kind really is.
> The code mentioned that it is defined in Constants.java, and I looked
> that up, but it is apparent not the right file. Maybe I am pointing to
> a wrong directory.
>
>
> StandardTokenizer tokenizer = new StandardTokenizer( r ); // r is a reader
> int count = 0;
> Token t = tokenizer .getNextToken();
>
> while (t != null) {
> count++;
> //if (t.kind == 0)
> // break;
> System.out.println( t );
> t = tokenizer.getNextToken() ;
> System.out.println(count);
> }
> System.out.println ( "done");
> Thank you for any input.
>
