[issue25643] Python tokenizer rewriting

Change by Pablo Galindo Salgado : -- pull_requests: +23830 pull_request: https://github.com/python/cpython/pull/25080 ___ Python tracker ___ ___

[issue25643] Python tokenizer rewriting

STINNER Victor added the comment: Oh, 6 years to fix this bug. Better late than never ;-) Thanks for reporting and for fixing it! -- ___ Python tracker ___ __

[issue25643] Python tokenizer rewriting

Change by Pablo Galindo Salgado : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ ___

[issue25643] Python tokenizer rewriting

Pablo Galindo Salgado added the comment: New changeset 261a452a1300eeeae1428ffd6e6623329c085e2c by Pablo Galindo in branch 'master': bpo-25643: Refactor the C tokenizer into smaller, logical units (GH-25050) https://github.com/python/cpython/commit/261a452a1300eeeae1428ffd6e6623329c085e2c -

[issue25643] Python tokenizer rewriting

Change by Pablo Galindo Salgado : -- keywords: +patch nosy: +pablogsal nosy_count: 7.0 -> 8.0 pull_requests: +23799 stage: -> patch review pull_request: https://github.com/python/cpython/pull/25050 ___ Python tracker

[issue25643] Python tokenizer rewriting

Brett Cannon added the comment: @serhiy: did you still want to commit this? -- ___ Python tracker ___ ___ Python-bugs-list mailing

[issue25643] Python tokenizer rewriting

Changes by Serhiy Storchaka : -- keywords: -patch versions: +Python 3.7 -Python 3.6 ___ Python tracker ___ ___ Python-bugs-list maili

[issue25643] Python tokenizer rewriting

Changes by Jim Fasarakis-Hilliard : -- nosy: +Jim Fasarakis-Hilliard ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscr

[issue25643] Python tokenizer rewriting

Roundup Robot added the comment: New changeset 23a7481eafd4 by Serhiy Storchaka in branch 'default': Issues #25643, #26581: Added new tests for detecting Python source code encoding. https://hg.python.org/cpython/rev/23a7481eafd4 -- nosy: +python-dev ___

[issue25643] Python tokenizer rewriting

Changes by Serhiy Storchaka : -- dependencies: +Double coding cookie ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscr

[issue25643] Python tokenizer rewriting

STINNER Victor added the comment: "especially for issues with null byte" I don't think that we should put to much energy in handling correctly NUL bytes. I see NUL bytes in code as bugs in the code, not in the Python parser. We *might* try to give warnings or better error messages to the user,

[issue25643] Python tokenizer rewriting

Stéphane Wirtel added the comment: Hi Serhiy, Just of your information but I think you know that, the tests pass ;-) [398/399] test_multiprocessing_spawn (138 sec) -- running: test_tools (108 sec) [399/399] test_tools (121 sec) 385 tests OK. 3 tests altered the execution environment: test__

[issue25643] Python tokenizer rewriting

Changes by Stéphane Wirtel : -- nosy: +matrixise ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.p

[issue25643] Python tokenizer rewriting

Changes by Brett Cannon : -- nosy: +brett.cannon ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.p

[issue25643] Python tokenizer rewriting

Changes by Yury Selivanov : -- nosy: +yselivanov ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.p

[issue25643] Python tokenizer rewriting

New submission from Serhiy Storchaka: Here is preliminary patch that refactors the lowest level of Python tokenizer, reading and decoding. It splits the code on smaller simpler functions, decreases the source size by 37 lines, and fixes bugs: issue14811, issue18961, and a number of others. Add