Hello, first of all thanks to everyone for replies and suggestions. I solved my problem by adapting the StandardTokenizer.jj and compiling it using javacc.
I replaced line 90: <ALPHANUM: (<LETTER>|<DIGIT>)+ > with <ALPHANUM: (<LETTER>|<DIGIT>|"_")+ > so that underscore is treated like alphanumeric characters. In my first tests, it seems to work perfectly. Anyhow, the problem remains that I can't understand how the described bevaviour might be the expected behaviour. I couldn't find the appropriate documentation in the javacc source of the tokenizer either. I suppose the source of the problem with underscores lies in the definition of NUM (floating point, serial, model numbers, ip addresses, etc.). No matter what, I guess my problem is solved. Thanks again and regards Sebastian -----Original Message----- From: Aigner, Thomas [mailto:[EMAIL PROTECTED] Sent: Tuesday, August 30, 2005 12:12 AM To: java-user@lucene.apache.org Subject: RE: Inconsistent tokenizing of words containing underscores. What seems to be working for me is a punctuation filter that removes / - _ etc and makes the token without them. Then "most" of the time the word XYZZZY_DE_SA0001 will be tokenized as XYZZZYDESA0001. For this to work, you will have to use the same punctuation filter on the strings before you search for them. Tom -----Original Message----- From: Daniel Naber [mailto:[EMAIL PROTECTED] Sent: Monday, August 29, 2005 3:15 PM To: java-user@lucene.apache.org Subject: Re: Inconsistent tokenizing of words containing underscores. On Monday 29 August 2005 19:21, Jeremy Meyer wrote: > The expected behavior is to sometimes treat a character as indicating a > new token and other times to ignore the same character? It depends on whether there are digits in the token. It's documented in the javacc source for the tokenizer(?). Regards Daniel -- http://www.danielnaber.de --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
