Nick Coghlan <ncogh...@gmail.com> added the comment:
Possible approach (untested):
def get_tokens(source):
if hasattr(source, "encode"):
# Already decoded, so bypass encoding detection
return _tokenize(io.StringIO(source).readline, None)
# Otherwise attempt to detect the correct encoding
return tokenize(io.BytesIO(source).readline)
----------
nosy: +ncoghlan
_______________________________________
Python tracker <rep...@bugs.python.org>
<http://bugs.python.org/issue9969>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
