Bart Lateur wrote:
>
> On Thu, 14 Sep 2000 15:47:43 -0700, Steve Fink wrote:
>
> >Currently, toke.c turns "foo$bar" into "foo".$bar before the parser or
> >anything else sees it. So any features implemented in the tokenizer have
> >to get smarter about remembering what they did.
>
> This sound pretty much like the same problem you face when designing a
> source level debugger, for any compiled language.
Yep. And they all punt, to varying degrees.
This case is probably most similar to C's preprocessor. And it totally
punts, by allowing macros to expand to one line only and not bothering
to trace things to the original code. The first is fine for us too (as
long as nobody wants byte positions reported); the second is not so
good.
I fully agree with the complaint about "use of uninitialized value in
concatenation (.)" -- the "(.)" in there especially adds insult to
injury. It's helpfully explaining what it means by 'concatenation' in
terms of your code, only that isn't your code.
But implementing this may mean associating an 'artificial?' flag to
every byte in the tokenizer. And checking it constantly. Or changing the
tokenizer to only pushback full tokens. Or... well, lots of
possibilities, but it definitely complicates things.
