[il-antlr-interest: 23749] Re: [antlr-interest] Lexer matching non-matching rule

[Jim Idle]

Micha wrote: On Saturday 16 May 2009 12:52:13 Jesper Larsson wrote: On Sat, 2009-05-16 at 08:27 +0530, Indhu Bharathi wrote: This is because on seeing 'f' of foo lexer has two options - 1. IDENT 2. URL. And it takes the second options since that seems to be longer that the first alternative. Note that the lexer always tries to match the longest token possible. but it fails to do so here. It isn't as cut and dried as the longest match - it is trying to make sense of your ambiguity basically, so when you don't get what you want, you have to guide it with more information. But, it isn't context sensitive so remember that the lexer runs first, then the parser runs with the results. I can understand the motivation of this restriction in the interest of keeping the lexer target code at a certain complexity level, but I have not seen it stated in the documentation. It would have been nice if the lexer generator had issued a warning. right. I don't like that behavior too, maybe it's because the antlr lexer is similar to a recursive descent parser. I think it should just return the longest match. Then it would not be an LL recognizer but just a pattern matcher like lex. You can always use a different lexer, but in practice once you get the hang of where you need to direct it, it isn't particularly complicated. If you want the longest match, then left factor everything and let it do that: A ( B (C|) |) ; And set the token type at the appropriate points. Jim --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "il-antlr-interest" group. To post to this group, send email to il-antlr-interest@googlegroups.com To unsubscribe from this group, send email to il-antlr-interest+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/il-antlr-interest?hl=en -~----------~----~----~----~------~----~------~--~---

List: http://www.antlr.org/mailman/listinfo/antlr-interest
Unsubscribe: 
http://www.antlr.org/mailman/options/antlr-interest/your-email-address