> > Well, not really. > > Technically, tokens are used to drive a parser from a scanner. > > If an AST node knows how to relate itself to its original source code > chunk and is able to print itself correctly, then tokens are redundant. > > In short, if you work with parsers, you'd better know what tokens are. If > you're only working with the AST, tokens are redundant and noise (i.e. they > often have a type (or more than one) which is only understood by the > parser). > > I'm working with ASTs sourceInterval. Trying to calculate it after method replaceWith:. You see, my proble was that each node of AST doesn't hold its start and stop position in same place. So I thought that token is such a place, however, eventually I understood that RBValueNodes don't have tokens:)
> Example of how it is done: > > RBPragmaNode > accessing-tokens gives access to left and right, which are > positions, not tokens. > Yes, I know. > > Thierry > > >> Mark >> >> >> >> 2014-10-28 11:59 GMT+02:00 Nicolai Hess <nicolaih...@web.de >> <mailto:nicolaih...@web.de>>: >> >> 2014-10-27 19:36 GMT+01:00 Mark Rizun <mri...@gmail.com >> <mailto:mri...@gmail.com>>: >> >> Hi all, >> >> Trying to understand here how tokens are used in AST. >> So far I can not see any order in usage of tokens. >> For instance, why RBValueNode doesn't have token? Is it haow >> it's supposed to be? >> >> >> RBValueNode is an abstract class. >> >> >> Cheers, >> Mark >> >> >> >> > >
