[issue9974] tokenizer.untokenize not invariant with line continuations

Terry J. Reedy added the comment: I added 5-tuple mode to roundtrip() in #20750. I solved the ENDMARKER problem by breaking out of the token loop if and when it appears. Reconstructing trailing whitespace other than \n is hopeless. The roundtrip test currently only tests equality of token sequ

[issue9974] tokenizer.untokenize not invariant with line continuations

Roundup Robot added the comment: New changeset 0f0e9b7d4f1d by Terry Jan Reedy in branch '2.7': Issue #9974: When untokenizing, use row info to insert backslash+newline. http://hg.python.org/cpython/rev/0f0e9b7d4f1d New changeset 24b4cd5695d9 by Terry Jan Reedy in branch '3.3': Issue #9974: When

[issue9974] tokenizer.untokenize not invariant with line continuations

Terry J. Reedy added the comment: The \ continuation bug is one of many covered by #12691 and its patch, but this came first and it focused on only this bug. With respect to this issue, the code patches are basically the same; I will use tests to choose between them. On #12691, Gareth notes th

[issue9974] tokenizer.untokenize not invariant with line continuations

Changes by Terry J. Reedy : -- assignee: -> terry.reedy ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https:/

[issue9974] tokenizer.untokenize not invariant with line continuations

Terry J. Reedy added the comment: One could argue that "The guarantee applies only to the token type and token string as the spacing between tokens (column positions) may change." covers merging of lines, but total elimination of needed whitespace is definitely a bug. -- nosy: +terry.

[issue9974] tokenizer.untokenize not invariant with line continuations

Dwayne Litzenberger added the comment: @amk: I'd appreciate it if you did. :) I ran into this bug while writing some code that converts b"..." into "..." in PyCrypto's setup.py script (for backward compatibility with Python 2.5 and below). -- nosy: +DLitz

[issue9974] tokenizer.untokenize not invariant with line continuations

Changes by Meador Inge : -- nosy: +meador.inge ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.pytho

[issue9974] tokenizer.untokenize not invariant with line continuations

Changes by Simon Law : -- nosy: +sfllaw ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/m

[issue9974] tokenizer.untokenize not invariant with line continuations

Changes by Eric Snow : -- nosy: +eric.snow ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.or

[issue9974] tokenizer.untokenize not invariant with line continuations

A.M. Kuchling added the comment: I looked at this a bit and made a revised version of the patch that doesn't add any line continuations when the token is ENDMARKER. It works on the example program and a few variations I tried, though I'm not convinced that it'll work for all possible permutat

[issue9974] tokenizer.untokenize not invariant with line continuations

Raymond Hettinger added the comment: > My patch handles the described situation, albeit a bit poorly ... Let us know when you've got a cleaned-up patch and have run the round-trip tests on a broad selection of files. For your test case, don't feel compelled to use doctest. It's okay to write

[issue9974] tokenizer.untokenize not invariant with line continuations

Brian Bossé added the comment: Yup, that's related to ENDMARKER being tokenized to its own line, even if EOF happens at the end of the last line of actual code. I don't know if anything relies on that behavior so I can't really suggest changing it. My patch handles the described situation, a

[issue9974] tokenizer.untokenize not invariant with line continuations

nick caruso added the comment: Additionally, substituting "a=1\n" for "a=1" results in no assertion and successful "untokenizing" to "a = 1\n" -- ___ Python tracker ___

[issue9974] tokenizer.untokenize not invariant with line continuations

nick caruso added the comment: -- import StringIO import tokenize tokens = [] def fnord(*a): tokens.append(a) tokenize.tokenize(StringIO.StringIO("a = 1").readline, fnord) tokenize.untokenize(tokens) -- Generates the same assertion

[issue9974] tokenizer.untokenize not invariant with line continuations

Kristján Valur Jónsson added the comment: Interesting, is that a separate defect of doctest? -- ___ Python tracker ___ ___ Python-bugs

[issue9974] tokenizer.untokenize not invariant with line continuations

Changes by Raymond Hettinger : -- assignee: -> rhettinger nosy: +rhettinger ___ Python tracker ___ ___ Python-bugs-list mailing list U

[issue9974] tokenizer.untokenize not invariant with line continuations

Brian Bossé added the comment: No idea if I'm getting the patch format right here, but tally ho! This is keyed from release27-maint Index: Lib/tokenize.py === --- Lib/tokenize.py (revision 85136) +++ Lib/tokenize.py (worki

[issue9974] tokenizer.untokenize not invariant with line continuations

Changes by Kristján Valur Jónsson : -- nosy: +krisvale ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://ma

[issue9974] tokenizer.untokenize not invariant with line continuations

New submission from Brian Bossé : Executing the following code against a py file which contains line continuations generates an assert: import tokenize foofile = open(filename, "r") tokenize.untokenize(list(tokenize.generate_tokens(foofile.readline))) (note, the list() is important due to issue