New submission from Edward K Ream <edream...@gmail.com>:
Tested on 3.6. tokenize.untokenize does not round-trip whitespace before backslash-newlines outside of strings: from io import BytesIO import tokenize # Round tripping fails on the second string. table = ( r''' print\ ("abc") ''', r''' print \ ("abc") ''', ) for s in table: tokens = list(tokenize.tokenize( BytesIO(s.encode('utf-8')).readline)) result = g.toUnicode(tokenize.untokenize(tokens)) print(result==s) I have an important use case that would benefit from a proper untokenize. After considerable study, I have not found a proper fix for tokenize.add_whitespace. I would be happy to work with anyone to rewrite tokenize.untokenize so that unit tests pass without fudges in TestRoundtrip.check_roundtrip. ---------- messages: 355827 nosy: edreamleo priority: normal severity: normal status: open title: Untokenize does not round-trip ws before bs-nl type: behavior versions: Python 3.6 _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue38663> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com