Could you elaborate on what you're trying to do, please?
Using StandardTokenizer in this low-level fashion is practically
unheard of, so I think knowing what you're attempting to do will help
us help you :)
Erik
On Sep 21, 2005, at 12:17 PM, Beady Geraghty wrote:
I see some definitions in StandardTokenizerConstants.java
Perhaps these are the values for t.kind.
Perhaps, I was confused between between the usage of
getNextToken() and next() in the standard analyzer.
When should one use getNextToken() instead of next()
I am just starting to use Lucene, so please excuse these
simple questions.
Thanks
On 9/21/05, Beady Geraghty <[EMAIL PROTECTED]> wrote:
Could someone tell me how to use the StandardTokenizer properly ?
I thought that if the tokenizer.getNextToken() returns null, then
it is
the end of stream. I have a loop that tries to get the next token
until
it is null. But the loop doesn't terminate.
I tried to termintae the loop by t.kind == 0, and it seems to have
stopped
upon the end of stream. I am not sure what t.kind really is.
The code mentioned that it is defined in Constants.java, and I looked
that up, but it is apparent not the right file. Maybe I am
pointing to
a wrong directory.
StandardTokenizer tokenizer = new StandardTokenizer( r ); // r is
a reader
int count = 0;
Token t = tokenizer .getNextToken();
while (t != null) {
count++;
//if (t.kind == 0)
// break;
System.out.println( t );
t = tokenizer.getNextToken() ;
System.out.println(count);
}
System.out.println ( "done");
Thank you for any input.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
