Re: tokenizing text using language analyzer but preserving stopwords if possible

2011-12-11 Thread KARTHIK SHIVAKUMAR
Hi >> tokenize the original foreign text into words Need to Identify the Appropriate analyzer ( foreign language before Indexing ...) with regards karthik On Wed, Dec 7, 2011 at 4:57 PM, Avi Rosenschein wrote: > On Wed, Dec 7, 2011 at 00:41, Ilya Zavorin wrote: > > > I need to implement a "

Re: tokenizing text using language analyzer but preserving stopwords if possible

2011-12-07 Thread Avi Rosenschein
On Wed, Dec 7, 2011 at 00:41, Ilya Zavorin wrote: > I need to implement a "quick and dirty" or "poor man's" translation of a > foreign language document by looking up each word in a dictionary and > replacing it with the English translation. So what I need is to tokenize > the original foreign te