Re: StopAnalyzer and apostrophes

2006-04-06 Thread Marvin Humphrey
On Apr 6, 2006, at 4:23 PM, Daniel Noll wrote: Marvin Humphrey wrote: I wrote: It looks like StopAnalyzer tokenizes by letter, and doesn't handle apostrophes. So, the input "I don't know" produces these tokens: don t know Is that right? It's not right. StopAnalyzer does to

Re: StopAnalyzer and apostrophes

2006-04-06 Thread Daniel Noll
Marvin Humphrey wrote: I wrote: It looks like StopAnalyzer tokenizes by letter, and doesn't handle apostrophes. So, the input "I don't know" produces these tokens: don t know Is that right? It's not right. StopAnalyzer does tokenize letter by letter, but 't' is a stopword, s

Re: StopAnalyzer and apostrophes

2006-04-06 Thread Marvin Humphrey
I wrote: It looks like StopAnalyzer tokenizes by letter, and doesn't handle apostrophes. So, the input "I don't know" produces these tokens: don t know Is that right? It's not right. StopAnalyzer does tokenize letter by letter, but 't' is a stopword, so the tokens are:

StopAnalyzer and apostrophes

2006-04-06 Thread Marvin Humphrey
Greets, It looks like StopAnalyzer tokenizes by letter, and doesn't handle apostrophes. So, the input "I don't know" produces these tokens: don t know Is that right? Marvin Humphrey Rectangular Research http://www.rectangular.com/ -