New submission from Gregory P. Smith <g...@krypto.org>:
The behavior change introduced in 3.6.7 and 3.7.1 via https://bugs.python.org/issue33899 has further consequences: ```python >>> tokenize.untokenize(tokenize.generate_tokens(io.StringIO('#').readline)) Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".../cpython/cpython-upstream/Lib/tokenize.py", line 332, in untokenize out = ut.untokenize(iterable) File ".../cpython/cpython-upstream/Lib/tokenize.py", line 266, in untokenize self.add_whitespace(start) File ".../cpython/cpython-upstream/Lib/tokenize.py", line 227, in add_whitespace raise ValueError("start ({},{}) precedes previous end ({},{})" ValueError: start (1,1) precedes previous end (2,0) ``` The same goes for using the documented tokenize API (`generate_tokens` is not documented): ``` tokenize.untokenize(tokenize.tokenize(io.BytesIO(b'#').readline)) ... ValueError: start (1,1) precedes previous end (2,0) ``` `untokenize()` is no longer able to work on output of `generate_tokens()` if the input to generate_tokens() did not end in a newline. Today's workaround: Always append a newline if one is missing to the line that the readline callable passed to tokenize or generate_tokens returns. Very annoying to implement. ---------- messages: 328876 nosy: ammar2, gregory.p.smith, meador.inge, taleinat, terry.reedy priority: normal severity: normal status: open title: untokenize() fails on tokenize output when a newline is missing versions: Python 3.6, Python 3.7, Python 3.8 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue35107> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
