On Jun 28, 1:46 am, Tim Roberts <t...@probo.com> wrote:
> Jim <cook_...@yahoo.com> wrote:
> >I'm trying to understand the output of the tokenize.generate_tokens()
> >generator. The token types returned seem to be more general than I'd
> >expect. For example, when fed the following line of code:
>
> >def func_a():
> >...
> >It seems to me that the token '(' should be identified as 'LPAR' and
> >')' as 'RPAR', as found in the dictionary token.tok_name. What am I
> >missing here?
>
> Did you read the module? Right at the top, it says:
>
> <quote>
> It is designed to match the working of the Python tokenizer exactly, except
> that it produces COMMENT tokens for comments and gives type OP for all
> operators
> </quote>
> --
> Tim Roberts, t...@probo.com
> Providenza & Boekelheide, Inc.
Thanks for the reply. I wonder why the tokenizer classifies all
operators simply as OP, instead of the various operators listed in the
tok_name dictionary.
Jim Cook
--
http://mail.python.org/mailman/listinfo/python-list
