[issue33338] [lib2to3] Synchronize token.py and tokenize.py with the standard library

Change by Irit Katriel : -- resolution: -> wont fix stage: patch review -> resolved status: open -> closed superseder: -> Close 2to3 issues and list them here ___ Python tracker

[issue33338] [lib2to3] Synchronize token.py and tokenize.py with the standard library

Change by monson : -- pull_requests: +8757 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.pytho

[issue33338] [lib2to3] Synchronize token.py and tokenize.py with the standard library

Łukasz Langa added the comment: I agree with you Serhiy, there's a number things I want to make faster. But first I'd like to merge implementations so there is a clear one-way diff ("this is what we updated in lib2to3 to make it consistent it Lib/tokenize.py"). Then I want to optimize.

[issue33338] [lib2to3] Synchronize token.py and tokenize.py with the standard library

Serhiy Storchaka added the comment: It seems to me that regular expressions used in the lib2to3 version are more efficient but more complex. $ ./python -m timeit -s 'import re; p = re.compile(r"0[bB](?:_?[01])+"); s = "0b"+"_0101"*16' 'p.match(s)' 10 loops, best of 5: 2.45 usec per loop

[issue33338] [lib2to3] Synchronize token.py and tokenize.py with the standard library

Łukasz Langa added the comment: New changeset c2d384dbd7c6ed9bdfaac45f05b463263c743ee7 by Łukasz Langa in branch 'master': bpo-8: [tokenize] Minor code cleanup (#6573) https://github.com/python/cpython/commit/c2d384dbd7c6ed9bdfaac45f05b463263c743ee7 -- __

[issue33338] [lib2to3] Synchronize token.py and tokenize.py with the standard library

Change by Łukasz Langa : -- pull_requests: +6274 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.

[issue33338] [lib2to3] Synchronize token.py and tokenize.py with the standard library

Change by Łukasz Langa : -- keywords: +patch pull_requests: +6269 stage: -> patch review ___ Python tracker ___ ___ Python-bugs-list

[issue33338] [lib2to3] Synchronize token.py and tokenize.py with the standard library

New submission from Łukasz Langa : lib2to3's token.py and tokenize.py were initially copies of the respective files from the standard library. They were copied to allow Python 3 to read Python 2's grammar. Since 2006, lib2to3 grew to be widely used as a Concrete Syntax Tree, also for parsing Py

[issue33338] [lib2to3] Synchronize token.py and tokenize.py with the standard library

Łukasz Langa added the comment: ### Diff between files The unified diff between tokenize implementations is here: https://gist.github.com/ambv/679018041d85dd1a7497e6d89c45fb86 It clocks at 275 lines but that's because it gives context. The actual diff is 175 lines long. To make it that small,