Edward K Ream <edream...@gmail.com> added the comment:

This post
https://groups.google.com/d/msg/leo-editor/DpZ2cMS03WE/VPqtB9lTEAAJ
discusses a complete rewrite of tokenizer.untokenize.

To quote from the post:

I have "discovered" a spectacular replacement for Untokenizer.untokenize in 
python's tokenize library module. The wretched, buggy, and impossible-to-fix 
add_whitespace method is gone. The new code has no significant 'if' statements, 
and knows almost nothing about tokens!  This is the way untokenize is written 
in The Book.

The new code should put an end to a long series of issues against untokenize 
code in python's tokenize library module.  Some closed issues were blunders 
arising from dumbing-down the TestRoundtrip.check_roundtrip method in 
test_tokenize.py. 

Imo, the way is now clear for proper unit testing of python's Untokenize class.

----------

_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue38663>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Reply via email to