You can also try using a state machine using the approach described in https://cs.brown.edu/~sk/Publications/Papers/Published/sk-automata-macros/paper.pdf On Feb 22, 2016 4:46 PM, "Federico Ramírez" <[email protected]> wrote:
> Hello everyone! I'm new to Scheme, and I need some help wrapping my head > around it. > > I want to write a simple tokenizer, which is just a function that takes a > string as input and outputs a list of "tokens", which to keep thing simple > is a tuple of NAME and VALUE. > > So, for example: > > (tokenize "foo = 1") => '((IDENTIFIER . "foo") (EQUALS . "=") (NUMBER . 1)) > > I didn't want to use global variables to keep things functional, so my > solution doesn't use any, but I wouldn't mind much using them. This is some > code for my current implementation, I omit some code for brevity as it's > quite repetitive for now: > > (define (match-identifier input) > (regexp-match #rx"^[a-zA-Z_]+")) > > (define (consume-identifier input) > (let ([match (first (match-identifier input))]) > (cons `(IDENTIFIER . ,match) (tokenize (substring input (string- > length match)))))) > > ;; some more matchers and consumers, all pretty similar > ;; ... > > (define (tokenize input) > (cond > ((match-identifier input) (consume-identifier input)) > ((match-equals input) (consume-equals input)) > ((match-number input) (consume-number input)) > (else '()))) > > What bothers me is that it's calling the matchers twice for each token, > which isn't very good for performance and it's not pretty :p > > I want to get rid of that double-check. I figured I could use some globals > in order to do it but I'd rather not. > > Does anyone have a suggestion/fix on my implementation? > > Thank you for your time :) > > -- > You received this message because you are subscribed to the Google Groups > "Racket Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Racket Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
