> On Oct 22, 2015, at 5:20 PM, Matt Wette <matthew.we...@verizon.net> wrote:
>
>
>> On Oct 18, 2015, at 8:53 PM, Nala Ginrut <nalagin...@gmail.com
>> <mailto:nalagin...@gmail.com>> wrote:
>> And I'm grad that you write a new lexer generator (before it I only know
>> silex), it's great! Would you like to make the generated tokens
>> compatible with scm-lalr? If so, people may rewrite their lexer module
>> with your lexer generator, and no need to rewrite the parser. I saw the
>> token name is string rather than symbol, so I guess it's not compatible
>> with scm-lalr.
>
> Actually, the lexer-generator uses convention of internally turning certain
> lexemes, like strings, into symbols like ‘$string, or integers into ‘$fixed.
> The argument to the lexer-generator is a “match-table” which says how to map
> the read items quoted items are identifiers (e.g., “while”) or character
> sequences (e.g., “+=“) to something the parser wants to see. For example, if
> you use the symbol WHILE to denote the source text “while” then you would
> have an entry (“while” . ‘WHILE) in the match table. So I think the
> lexer-generator should be adaptable to other parsers.
I didn’t describe this very well. I will try again. The code actually
provides a lexical analyzer (aka lexer) generator-generator. To make a lexer
you call make-lexer-generator with a match-table as argument:
(define gen-lexer (make-lexer-generator match-table))
Then when you pass a generated lexer each time you call the parser:
(parse (gen-lexer))
The reason is that the lexer keeps state information (e.g., the
beginning-of-line condition). Now the match table argument indicates how the
user wants lexemes, read from the input, to be reported to the parser. If you
want “while” in the input to be reported as ‘WHILE to the parser, then the
match table would include an entry ‘(“while” . WHILE). The generator uses
special symbols to represent quoted strings, numbers and comments. If you want
quoted strings returned with the symbol ‘STRING, then the match table would
include an entry ‘($string . STRING).
In many cases I have nyacc "hashify” my parser so that it uses integers instead
of symbols. Here is the match table generated for the hashified matlab parser:
(define mtab
'(($lone-comm . 1) ($string . 2) ($float . 3) ($fixed . 4) ($ident . 5) (
";" . 6) (".'" . 7) ("'" . 8) ("~" . 9) (".^" . 10) (".\\" . 11) ("./" .
12) (".*" . 13) ("^" . 14) ("\\" . 15) ("/" . 16) ("*" . 17) ("-" . 18) (
"+" . 19) (">=" . 20) ("<=" . 21) (">" . 22) ("<" . 23) ("~=" . 24) ("=="
. 25) ("&" . 26) ("|" . 27) (":" . 28) ("case" . 29) ("elseif" . 30) (
"clear" . 31) ("global" . 32) ("return" . 33) ("otherwise" . 34) ("switch"
. 35) ("else" . 36) ("if" . 37) ("while" . 38) ("for" . 39) ("," . 40) (
")" . 41) ("(" . 42) ("=" . 43) ("]" . 44) ("[" . 45) ("function" . 46) (
#\newline . 47) ("end" . 48) ($end . 49)))
and here is the match table generated for the non-hashified match table for the
same language:
(define mtab
'(($lone-comm . $lone-comm) ($string . $string) ($float . $float) ($fixed
. $fixed) ($ident . $ident) (";" . #{$:;}#) (".'" . $:.') ("'" . $:') ("~"
. $:~) (".^" . $:.^) (".\\" . $:.\) ("./" . $:./) (".*" . $:.*) ("^" .
$:^) ("\\" . $:\) ("/" . $:/) ("*" . $:*) ("-" . $:-) ("+" . $:+) (">=" .
$:>=) ("<=" . $:<=) (">" . $:>) ("<" . $:<) ("~=" . $:~=) ("==" . $:==) (
"&" . $:&) ("|" . $:|) (":" . $::) ("case" . $:case) ("elseif" . $:elseif)
("clear" . $:clear) ("global" . $:global) ("return" . $:return) (
"otherwise" . $:otherwise) ("switch" . $:switch) ("else" . $:else) ("if"
. $:if) ("while" . $:while) ("for" . $:for) ("," . $:,) (")" . #{$:\x29;}#
) ("(" . #{$:\x28;}#) ("=" . $:=) ("]" . #{$:\x5d;}#) ("[" . #{$:\x5b;}#)
("function" . $:function) (#\newline . #\newline) ("end" . $:end) ($end .
$end)))
