Dan Sugalski wrote:
>
> At 04:16 PM 11/28/00 -0800, Steve Fink wrote:
> >Perl5 is parseable with a single token of lookahead and lots of
> >parser/lexer communication. Sort of. It would be nice to prevent it from
> >getting any worse.
>
> I'm really thinking that the lexer, parser, and tokenizer can't be anywhere
> near as separate as we'd like. I think we're going to end up with a rather
> odd mutant beast. Hopefully one that's understandable by reasonably sane
> people...
With the PL_expect stuff and recursive calls in there, I'd hardly call
them separable now. So I have no argument with what you said, but I do
if you s/as we'd like/as it is now/.
I think there are big advantages in keeping the parsing and tokenization
(what's the difference between a lexer and a tokenizer, btw?) logically
as separate as possible, but no more. It's a very useful separation for
keeping things clean and understandable. It just has one minor drawback
-- it doesn't actually work. Not for perl, at least, and not for a whole
lot of other languages too. But weakening the separation enough to make
things possible and hopefully relatively easy shouldn't have to mean
rolling them together completely.
On the other hand, I don't really mean I don't want it to get any closer
together than it is now. We do need a higher bandwidth of communication
between the two. Right now, it's pretty much a shiny clean byacc parser
resting atop a writhing intestinal sea of tokenization code. The parser
really ought to be shouldering more of the burden. Please don't think
too much about the bizarre mixture of those metaphors.
