Re: [racket-users] Help writing a simple lexer/tokenizer

2016-02-22 Thread Jon Zeppieri
On Mon, Feb 22, 2016 at 7:26 PM, Neil Van Dyke wrote: > For someone who really wants to learn this well, rather than just make a > lexer and move on... > > I started learning the pertinent theory (such as automata NFA/DFA, and > classes of formal languages) from the original red dragon book: > ht

Re: [racket-users] Help writing a simple lexer/tokenizer

2016-02-22 Thread Neil Van Dyke
For someone who really wants to learn this well, rather than just make a lexer and move on... I started learning the pertinent theory (such as automata NFA/DFA, and classes of formal languages) from the original red dragon book: https://en.wikipedia.org/wiki/Compilers:_Principles,_Techniques,_

Re: [racket-users] Help writing a simple lexer/tokenizer

2016-02-22 Thread Justin Zamora
You can also try using a state machine using the approach described in https://cs.brown.edu/~sk/Publications/Papers/Published/sk-automata-macros/paper.pdf On Feb 22, 2016 4:46 PM, "Federico Ramírez" wrote: > Hello everyone! I'm new to Scheme, and I need some help wrapping my head > around it. > >

Re: [racket-users] Help writing a simple lexer/tokenizer

2016-02-22 Thread Federico Ramírez
On Monday, 22 February 2016 19:25:50 UTC-3, Jon Zeppieri wrote: > Have you looked at the parser-tools/lex library? > https://docs.racket-lang.org/parser-tools/Lexers.html > > > On Feb 22, 2016, at 4:46 PM, Federico Ramírez wrote: > > > > Hello everyone! I'm new to Scheme, and I need some help

Re: [racket-users] Help writing a simple lexer/tokenizer

2016-02-22 Thread Federico Ramírez
On Monday, 22 February 2016 19:34:40 UTC-3, Jens Axel Søgaard wrote: > 2016-02-22 22:46 GMT+01:00 Federico Ramírez : > > >     (define (tokenize input) > >       (cond > >         ((match-identifier input) (consume-identifier input)) > >         ((match-equals     input) (consume-equals     i

Re: [racket-users] Help writing a simple lexer/tokenizer

2016-02-22 Thread Jens Axel Søgaard
2016-02-22 22:46 GMT+01:00 Federico Ramírez : (define (tokenize input) > (cond > ((match-identifier input) (consume-identifier input)) > ((match-equals input) (consume-equals input)) > ((match-number input) (consume-number input)) > (else '

Re: [racket-users] Help writing a simple lexer/tokenizer

2016-02-22 Thread Jon Zeppieri
Have you looked at the parser-tools/lex library? https://docs.racket-lang.org/parser-tools/Lexers.html > On Feb 22, 2016, at 4:46 PM, Federico Ramírez wrote: > > Hello everyone! I'm new to Scheme, and I need some help wrapping my head > around it. > > I want to write a simple tokenizer, which

[racket-users] Help writing a simple lexer/tokenizer

2016-02-22 Thread Federico Ramírez
Hello everyone! I'm new to Scheme, and I need some help wrapping my head around it. I want to write a simple tokenizer, which is just a function that takes a string as input and outputs a list of "tokens", which to keep thing simple is a tuple of NAME and VALUE. So, for example: (tokenize "fo