Matthias Bussonnier <bussonniermatth...@gmail.com> added the comment:
> Why not just bless the existing generate_tokens() function as a public API, Yes please, or just make the private `_tokenize` public under another name. The `tokenize.tokenize` method try to magically detect encoding which may be unnecessary. ---------- nosy: +mbussonn _______________________________________ Python tracker <rep...@bugs.python.org> <https://bugs.python.org/issue12486> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
