Re: How to tokenize with comma in standard tokenizer

2007-09-18 Thread Bhavin Pandya
Thanks mark. Take the comma out of: | <#P: ("_"|"-"|"/"|"."|",") > in the .jj file Its working for me... - Bhavin pandya - Original Message - From: "Mark Miller" <[EMAIL PROTECTED]> To: Sent: Monday, Septem

Re: How to tokenize with comma in standard tokenizer

2007-09-17 Thread Mark Miller
Take the comma out of: | <#P: ("_"|"-"|"/"|"."|",") > in the .jj file (around line 92). Keep in mind that this will affect being able to find tokens that where previously indexed with the comma there (obviously). I believe the javacc target in the build file will rebuild...you need to get javac

How to tokenize with comma in standard tokenizer

2007-09-17 Thread Bhavin Pandya
Hi, Standard tokenizer works pretty well for me... but i found one problem with my usage... I want to tokenize..."TheRing6,Proposal6,GuyandGirl6" as a three saparate tokens.. while standard analyzer considering it as a one word because it has one digit in token. Expected three tokens: 1. ther