Re: Extending StandardTokenizer Jflex to not split on '/'

2014-02-20 Thread Diego Fernandez
Thanks again for the help. Upon further investigation I found out we weren't using our custom version of the analyzer, which explains why it wasn't doing what I thought it should. When I have time to get back to it I'll reconfigure it to use our tokenizer. Diego Fernandez - 爱国 Software Engine

Re: Extending StandardTokenizer Jflex to not split on '/'

2014-02-17 Thread Steve Rowe
Sorry, Diego, the generated scanner diff doesn't tell me anything. Since I was able to successfully make changes to the open source and get the desired behavior, I'm guessing you're: a) not using the same (versions of) tools as me; b) not using the same (version of the) source as me; or c) not tes

Re: Extending StandardTokenizer Jflex to not split on '/'

2014-02-17 Thread Diego Fernandez
Hey Steve, thanks for the quick reply. I didn't have a chance to test again until today. In our Lucene build, we had already made some customization to the JFlex file and it re-generates the java file whenever we build our project. Unfortunately, it is still not working for me. I diffed the

Re: Extending StandardTokenizer Jflex to not split on '/'

2014-02-14 Thread Steve Rowe
Welcome Diego, I think you’re right about MidLetter - adding a char to it should disable splitting on that char, as long as there is a letter on one side or the other. (If you’d like that behavior to be extended to numeric digits, you should use MidNumLet instead.) I tested this by adding “/“