Larry Wall wrote : > > It's not clear that the lexer is a separate entity any more. Lexers > were originally invented as a way of abstracting out part of the > grammar so that it could be done in a separate pass, and to simplify > the grammar for the poor overworked parser.
Indeed. Another benefit of lexers is to allow contextual constructs, like the qq/.../ operator, where the separators can be an arbitrary character. (well, a non-contextual grammar can describe this, if you enumerate all possible separators in separate grammar productions, and you'll end up with a gigantic transition table.) > But you can write a > grammar for an identifier just about as easily as for an if-then-else. > More easily, if we're basing it on regexes. If we're viewing all > grammar through the lens of regexes, we're really starting out closer > to Lexerland anyway, and generalizing toward parsing, a traditionally > weaker area for Perl. And that's an odd weakness for a text > processing language. This sounds really cool. The word 'regular' goes more and more unappropriate...
