On Wed, May 10, 2006 at 11:25:26AM +1000, Damian Conway wrote: : True. "Token" is the wrong word for another reason: a token is a : segments component of the input stream, *not* a rule for matching : segmented components of the input stream. The correct term for that is : "terminal". So a suitable keyword might well be "term".
There are several problems with that. A small problem is that "term" is the same length as "rule", and that makes it harder to tell them apart visually. A larger problem is that, unfortunately, "term" is one of the more heavily overloaded terms (pun intended) in computing. Even in Perl 5 culture we use it *heavily* to mean "non-infix". Calling infix:<*> a "term" really grates for that reason. The overloading of "token" is much milder, and I'd rather take the core metaphor of token and extend it to the supertoken, because the intent is the same. The intent of a token is to present a simple interface outward. The same is true for the supertoken. Structurally a supertoken is rather like an object, insofar as it has a simple outside and a complicated inside. That complicated inside is expressed by the fact that the supertoken calls out to a subrule. But the supertoken itself still wants to be treated simply in its own context, just as any object can be treated as a scalar. The interface to a postcircumfix requires token parsing on the outside, despite allowing full expressions on the inside. But as with the sub/multi/method distinction, the primary motivation is to distinguish the outward interface, that is, how they are to be used. So anyway, I think "token" is sufficiently close to what we want it to mean that we can force it to mean that, and it's sufficiently orphaned that few people are going to complain about impressing it into forced labor. And, in fact, the larger cultural meaning of token implies that it's something simple that represents something complicated, as in "a token of our appreciation." Larry
