[issue1420] Unicode literals in tokenize.py and tests.

Guido van Rossum added the comment: Got it. Checked in as revision 58951. -- resolution: -> accepted status: open -> closed __ Tracker <[EMAIL PROTECTED]> __ _

[issue1420] Unicode literals in tokenize.py and tests.

Ron Adam added the comment: George is correct. The changes are minimal. The only addition is to run the tokenize_tests.txt file though compile() as a way to force an exception if it needs updating in the future. The results of the compile are not used. __ Track

[issue1420] Unicode literals in tokenize.py and tests.

Georg Brandl added the comment: Martin, Guido: I think you misunderstand the patch description: it doesn't make tokenize process bytes instead of bytes, but makes it tokenize the new b"..." literals instead of the old u"..." literals. __ Tracker <[EMAIL PROTECTED]

[issue1420] Unicode literals in tokenize.py and tests.

Guido van Rossum added the comment: I'm with Martin. Adam, why do you think tokenize should use bytes instead of text strings? -- nosy: +gvanrossum __ Tracker <[EMAIL PROTECTED]> __ __

[issue1420] Unicode literals in tokenize.py and tests.

Martin v. Löwis added the comment: I think this patch is wrong. Python source code is inherently text, so generate_tokens should decode the input, rather than operating on bytes. -- nosy: +loewis __ Tracker <[EMAIL PROTECTED]>

[issue1420] Unicode literals in tokenize.py and tests.

Christian Heimes added the comment: Yes, raw byte strings are possible: >>> br"\x" b'\\x' -- keywords: +patch, py3k nosy: +tiran priority: -> normal __ Tracker <[EMAIL PROTECTED]> __ ___

[issue1420] Unicode literals in tokenize.py and tests.

Georg Brandl added the comment: I don't think you can have raw bytes (rb"..." etc.) literals. -- nosy: +georg.brandl __ Tracker <[EMAIL PROTECTED]> __ _

[issue1420] Unicode literals in tokenize.py and tests.

New submission from Ron Adam: Replaced Unicode literals in tokenize.py and it's tests files with byte literals. Added a compile step to the test to make sure the text file used in the test are valid python code. This will catch changes that need to be done in to the text (gold file) for future