Serhiy Storchaka added the comment: Patch to allow tokenize() accepts string is very simple, only 4 lines. But it requires a lot of documentation changes.
Then we can get rid of undocumented generate_tokens(). Note, stdlib an tools use only generate_tokens(), none uses tokenize(). Of course, it will be better if tokenize() will work with iterator protocol. Here is a preliminary patch. I will be thankful for the help with the documentation and for the discussion. Of course, it will be better if tokenize() will work with iterator protocol. ---------- keywords: +patch nosy: +serhiy.storchaka versions: +Python 3.4 -Python 3.3 Added file: http://bugs.python.org/file27587/tokenize_str.diff _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue12486> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
