[issue13832] tokenization assuming ASCII whitespace; missing multiline case

2012-01-20 Thread Benjamin Peterson
Benjamin Peterson added the comment: $ ./python Python 3.3.0a0 (default:50a4af2ca654+, Jan 20 2012, 10:59:48) [GCC 4.5.3] on linux Type "help", "copyright", "credits" or "license" for more information. >>> s="""x=5 # comment ... x=6 ... """ >>> compile(s, "", "single") Traceback (most recent

[issue13832] tokenization assuming ASCII whitespace; missing multiline case

2012-01-20 Thread Jim Jewett
Jim Jewett added the comment: Ignoring non-ascii whitespace is defensible, and I agree that it should match the rest of the parser. Ignoring 2nd lines is still a problem, and supposedly part of what got fixed. Test case: s="""x=5 # comment x=6 """ compile(s, "", 'single') --

[issue13832] tokenization assuming ASCII whitespace; missing multiline case

2012-01-19 Thread Benjamin Peterson
Benjamin Peterson added the comment: The tokenizer doesn't consider unicode spaces, either. -- resolution: -> fixed status: open -> closed ___ Python tracker ___ __

[issue13832] tokenization assuming ASCII whitespace; missing multiline case

2012-01-19 Thread Roundup Robot
Roundup Robot added the comment: New changeset 00c4efbf57c3 by Benjamin Peterson in branch 'default': check after comments, too (#13832) http://hg.python.org/cpython/rev/00c4efbf57c3 -- nosy: +python-dev ___ Python tracker

[issue13832] tokenization assuming ASCII whitespace; missing multiline case

2012-01-19 Thread Antoine Pitrou
Changes by Antoine Pitrou : -- nosy: +benjamin.peterson ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://

[issue13832] tokenization assuming ASCII whitespace; missing multiline case

2012-01-19 Thread Jim Jewett
New submission from Jim Jewett : Parser/parsetok.c was recently changed (e.g. http://hg.python.org/cpython/rev/2bd7f40108b4 ) to raise an error if multiple statements were found in a single-statement compile call. It sensibly ignores trailing whitespace and comments. Unfortunately, (1) It