[issue2182] tokenize: does not allow CR for a newline

Jared Grubb <[EMAIL PROTECTED]> added the comment: I actually hadnt thought of that. PyPy should actually use universal newlines to its advantage; after all, it IS written in Python... Thanks for the suggestion! In any case, I wanted to get this bug about the standard library in your record, in

[issue2182] tokenize: does not allow CR for a newline

Guido van Rossum <[EMAIL PROTECTED]> added the comment: I recommend that you only care about \n and consider everything else unspecified. __ Tracker <[EMAIL PROTECTED]> __ _

[issue2182] tokenize: does not allow CR for a newline

Jared Grubb <[EMAIL PROTECTED]> added the comment: Yes, but exec(string) also gives a syntax error for \r\n: exec('x=1\r\nprint x') The only explanation I could find for ONLY permitting \n as newlines in exec(string) comes from PEP278: "There is no support for universal newlines in strings pa

[issue2182] tokenize: does not allow CR for a newline

Guido van Rossum <[EMAIL PROTECTED]> added the comment: I still think it shouldn't be tokenize's business to handle this. I'm not quite sure how exec() manages to do this; I note that this gives a syntax error: exec('x = 1\rprint x\r') __ Tracker <[EMAIL PROTECT

[issue2182] tokenize: does not allow CR for a newline

Jared Grubb <[EMAIL PROTECTED]> added the comment: This is not a report on a bug in exec(), but rather a bug in the tokenize module -- the behavior between the CPython tokenizer and the tokenize module is not consistent. If you look in the tokenize.py source, it contains code to recognize both \n

[issue2182] tokenize: does not allow CR for a newline

Guido van Rossum <[EMAIL PROTECTED]> added the comment: I don't think this ought to be changed in exec(). It ought to be done by opening the file using universal newlines. -- resolution: -> rejected status: open -> closed __ Tracker <[EMAIL PROTECTED]>

[issue2182] tokenize: does not allow CR for a newline

Changes by Sean Reifschneider <[EMAIL PROTECTED]>: -- assignee: -> gvanrossum components: +Library (Lib) -Extension Modules nosy: +gvanrossum priority: -> normal __ Tracker <[EMAIL PROTECTED]> _

[issue2182] tokenize: does not allow CR for a newline

New submission from Jared Grubb: tokenize recognizes '\n' and '\r\n' as newlines, but does not tolerate '\r': >>> s = "print 1\nprint 2\r\nprint 3\r" >>> open('temp.py','w').write(s) >>> exec(open('temp.py','r')) 1 2 3 >>> tokenize.tokenize(open('temp.py','r').readline) 1,0-1,5:NAME'