[issue2134] function generate_tokens at tokenize.py yields wrong token for colon

Terry J. Reedy added the comment: I have not looked at this, but a new parameter would be a new feature. Its a moot point until there is an agreed on patch for a current version. -- nosy: +tjreedy type: behavior -> feature request versions: +Python 3.2 -Python 2.4, Python 2.5, Python 2

[issue2134] function generate_tokens at tokenize.py yields wrong token for colon

A.M. Kuchling added the comment: Unfortunately I think this will break many users of tokenize.py. e.g. http://browsershots.googlecode.com/svn/trunk/devtools/pep8/pep8.py has code like: if (token_type == tokenize.OP and text in '([' and ...): If tokenize now returns LPAR, this code will no lon

[issue2134] function generate_tokens at tokenize.py yields wrong token for colon

Changes by Christian Heimes: -- keywords: +patch priority: -> normal __ Tracker <[EMAIL PROTECTED]> __ ___ Python-bugs-list mailing list Unsub

[issue2134] function generate_tokens at tokenize.py yields wrong token for colon

Guilherme Polo added the comment: I'm attaching a patch that solves this and updates tests. Added file: http://bugs.python.org/file9459/tokenize_r60884.diff __ Tracker <[EMAIL PROTECTED]> __ _

[issue2134] function generate_tokens at tokenize.py yields wrong token for colon

New submission from Guilherme Polo: function generate_tokes at tokenize.py yields token OP (51) for colon, while it should be token COLON (11). It probably affects other python versions as well. I'm attaching a minor sample that demonstrates this, running it returns the following output: 1 'if'