Re: Tokenizer

2020-03-19 Thread Andi Vajda
On Thu, 19 Mar 2020, Marc Jeurissen wrote: Pylucene version: 8.1.1 Hi all, When you have a custom tokenizer (class CustomTokenizer(PythonTokenizer)), you don?t seem to be able to override any method besides incrementToken (so not end, reset, close). Is this correct? Correct, the only

Tokenizer

2020-03-19 Thread Marc Jeurissen
Pylucene version: 8.1.1 Hi all, When you have a custom tokenizer (class CustomTokenizer(PythonTokenizer)), you don’t seem to be able to override any method besides incrementToken (so not end, reset, close). Is this correct? Thank you very much Met vriendelijke groeten, Marc Jeurissen

Re: Tokenizer text source

2016-10-27 Thread Andi Vajda
On Tue, 25 Oct 2016, Marc Jeurissen wrote: I have a custom Analyzer and Tokenizer which I'm trying to migrate from Pylucene 4.10 to 6.2. Problem is that it is no longer possible to grab the text source from neither the createComponents method or the Tokenizer constructor. Documentation

Tokenizer text source

2016-10-25 Thread Marc Jeurissen
Hi, I have a custom Analyzer and Tokenizer which I'm trying to migrate from Pylucene 4.10 to 6.2. Problem is that it is no longer possible to grab the text source from neither the createComponents method or the Tokenizer constructor. Documentation says the Tokenizer has a field &

Re: Building a custom Tokenizer

2010-07-18 Thread Martin
lyzer samples. Thanks again, Martin On Jul 17, 2010, at 22:30, Andi Vajda wrote: On Jul 17, 2010, at 22:23, Martin wrote: Hi there, I'm trying to extend the PythonTokenizer class to build my own custom tokenizer, but seem to get stuck pretty much soon after that. I know that I

Re: Building a custom Tokenizer

2010-07-18 Thread Andi Vajda
On Jul 17, 2010, at 22:30, Andi Vajda wrote: On Jul 17, 2010, at 22:23, Martin wrote: Hi there, I'm trying to extend the PythonTokenizer class to build my own custom tokenizer, but seem to get stuck pretty much soon after that. I know that I'm supposed to extend the incr

Re: Building a custom Tokenizer

2010-07-17 Thread Andi Vajda
On Jul 17, 2010, at 22:23, Martin wrote: Hi there, I'm trying to extend the PythonTokenizer class to build my own custom tokenizer, but seem to get stuck pretty much soon after that. I know that I'm supposed to extend the incrementToken() method, but what exactly am I deali

Building a custom Tokenizer

2010-07-17 Thread Martin
Hi there, I'm trying to extend the PythonTokenizer class to build my own custom tokenizer, but seem to get stuck pretty much soon after that. I know that I'm supposed to extend the incrementToken() method, but what exactly am I dealing with in there and what should it return? My g

Re: Ngram tokenizer

2009-10-29 Thread Andi Vajda
On Oct 29, 2009, at 11:48, abhinav mishra wrote: It gives an error. And what is the error ? May be the problem is with the way I invoked the function. And what is the way ? I think a sample code to tokenize(ngram) would be sufficient. I'm trying to understand if your problem is a Luc

Re: Ngram tokenizer

2009-10-29 Thread abhinav mishra
It gives an error. May be the problem is with the way I invoked the function. I think a sample code to tokenize(ngram) would be sufficient. Abhinav On Thu, Oct 29, 2009 at 11:26 AM, Andi Vajda wrote: > > On Oct 29, 2009, at 10:28, abhinav mishra wrote: > > Hi, >> >> I know that there is a cl

Re: Ngram tokenizer

2009-10-29 Thread Andi Vajda
On Oct 29, 2009, at 10:28, abhinav mishra wrote: Hi, I know that there is a class EdgeNGramTokenizer, which can be used to extract n-grams. However, I'm not able use it. What do you mean by "not able to use it" ? Andi.. Any pointers or maybe, piece of code in this regard would be helpful

Ngram tokenizer

2009-10-29 Thread abhinav mishra
Hi, I know that there is a class EdgeNGramTokenizer, which can be used to extract n-grams. However, I'm not able use it. Any pointers or maybe, piece of code in this regard would be helpful. Regards, Abhinav