Re: adding attributes to TokenStream

2013-01-01 Thread Michael Sokolov
Sure ... The frequency count is maintained in the index to enable relevance scoring. You can pull it out using a TermDocs, which enumerates this sort of information. Sorry, I don't have example code handy for this. -Mike On 1/1/2013 4:24 PM, Itai Peleg wrote: That worked great :) thanks a

Re: adding attributes to TokenStream

2013-01-01 Thread Itai Peleg
That worked great :) thanks a lot for the quick reply! I have another question - after I "flagged" all my special tokens (in my case, the ones that are entities) is there an elegant way of counting how many of them I have in a document? I found an ugly way to do that, but I'm sure there's a better

Re: adding attributes to TokenStream

2012-12-31 Thread Michael Sokolov
On 12/31/2012 11:39 AM, Itai Peleg wrote: Hi all, Can someone please post a simple example showing how to add additional attributes to token in a TokenStream (inside IncrementToken for example?). I'm working on entity extraction and want to flag specific tokens an entities, but I'm having probl