[issue12675] tokenize module happily tokenizes code with syntax errors

Irit Katriel added the comment: Reproduced on 3.11. -- nosy: +iritkatriel versions: +Python 3.11 -Python 2.7, Python 3.3, Python 3.4 ___ Python tracker ___ ___

[issue12675] tokenize module happily tokenizes code with syntax errors

Changes by Terry J. Reedy : -- versions: +Python 3.4 -Python 3.2 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue12675] tokenize module happily tokenizes code with syntax errors

Changes by Florent Xicluna : -- nosy: +flox ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue12675] tokenize module happily tokenizes code with syntax errors

Changes by Meador Inge : -- nosy: +meador.inge ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue12675] tokenize module happily tokenizes code with syntax errors

Changes by Sandro Tosi : -- nosy: +sandro.tosi ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pyth

[issue12675] tokenize module happily tokenizes code with syntax errors

Gareth Rees added the comment: Terry: agreed. Does anyone actually use this module? Does anyone know what the design goals are for tokenize? If someone can tell me, I'll do my best to make it meet them. Meanwhile, here's another bug. Each character of trailing whitespace is tokenized as an E

[issue12675] tokenize module happily tokenizes code with syntax errors

Terry J. Reedy added the comment: I have not used tokenize, but if it is *not* intended to exactly reproduce the internal tokenizer behavior, the claim that it is should be amended. -- nosy: +terry.reedy ___ Python tracker

[issue12675] tokenize module happily tokenizes code with syntax errors

Changes by Daniel Urban : -- nosy: +durban ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.o

[issue12675] tokenize module happily tokenizes code with syntax errors

Gareth Rees added the comment: Ah ... TokenInfo is a *subclass* of namedtuple, so I can add extra properties to it without breaking consumers that expect it to be a 5-tuple. -- ___ Python tracker

[issue12675] tokenize module happily tokenizes code with syntax errors

Gareth Rees added the comment: Having looked at some of the consumers of the tokenize module, I don't think my proposed solutions will work. It seems to be the case that the resynchronization behaviour of tokenize.py is important for consumers that are using it to transform arbitrary Python s

[issue12675] tokenize module happily tokenizes code with syntax errors

Gareth Rees added the comment: I'm having a look to see if I can make tokenize.py better match the real tokenizer, but I need some feedback on a couple of design decisions. First, how to handle tokenization errors? There are three possibilities: 1. Generate an ERRORTOKEN, resynchronize, and

[issue12675] tokenize module happily tokenizes code with syntax errors

Benjamin Peterson added the comment: tokenize has useful features that the builtin tokenizer does not possess such as the NL token. -- ___ Python tracker ___ __

[issue12675] tokenize module happily tokenizes code with syntax errors

Vlad Riscutia added the comment: How come tokenizer module is not based on actual C tokenizer? Wouldn't that make more sense (and prevent this kind of issues)? -- nosy: +vladris ___ Python tracker ___

[issue12675] tokenize module happily tokenizes code with syntax errors

Changes by Eric Snow : -- nosy: +ericsnow ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.or

[issue12675] tokenize module happily tokenizes code with syntax errors

Changes by Ezio Melotti : -- stage: -> test needed ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail

[issue12675] tokenize module happily tokenizes code with syntax errors

Changes by Ezio Melotti : -- nosy: +ezio.melotti versions: +Python 2.7, Python 3.2 ___ Python tracker ___ ___ Python-bugs-list mailing

[issue12675] tokenize module happily tokenizes code with syntax errors

Benjamin Peterson added the comment: This should probably be fixed (patches welcome). However, note even with valid Python code, the tokens are not the same. -- ___ Python tracker

[issue12675] tokenize module happily tokenizes code with syntax errors

Gareth Rees added the comment: These errors are generated directly by the tokenizer. In tokenizer.c, the tokenizer generates ERRORTOKEN when it encounters something it can't tokenize. This causes parsetok() in parsetok.c to stop tokenizing and return an error. --

[issue12675] tokenize module happily tokenizes code with syntax errors

R. David Murray added the comment: I'm not familiar with the parser internals (I'm nosying someone who is), but I suspect what you are seeing at the command line is the errors being caught at a stage later than the tokenizer. -- nosy: +benjamin.peterson, r.david.murray __

[issue12675] tokenize module happily tokenizes code with syntax errors

New submission from Gareth Rees : The tokenize module is happy to tokenize Python source code that the real tokenizer would reject. Pretty much any instance where tokenizer.c returns ERRORTOKEN will illustrate this feature. Here are some examples: Python 3.3.0a0 (default:2d69900c0820, Aug