[issue33766] Grammar Incongruence

Change by Terry J. Reedy : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___ __

[issue33766] Grammar Incongruence

Change by miss-islington : -- pull_requests: +7201 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://ma

[issue33766] Grammar Incongruence

miss-islington added the comment: New changeset f01b951a0e70f36ca2a3caa043f89a5277bb0bb0 by Miss Islington (bot) in branch '2.7': bpo-33766: Document that end of file or string is a newline (GH-7383) https://github.com/python/cpython/commit/f01b951a0e70f36ca2a3caa043f89a5277bb0bb0 -

[issue33766] Grammar Incongruence

Change by miss-islington : -- pull_requests: +7200 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://ma

[issue33766] Grammar Incongruence

Change by miss-islington : -- pull_requests: +7199 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://ma

[issue33766] Grammar Incongruence

Terry J. Reedy added the comment: New changeset 0aa17ee6a76df0946d42e7657a501f1862065a22 by Terry Jan Reedy (Ammar Askar) in branch 'master': bpo-33766: Document that end of file or string is a newline (GH-7383) https://github.com/python/cpython/commit/0aa17ee6a76df0946d42e7657a501f1862065a22

[issue33766] Grammar Incongruence

Terry J. Reedy added the comment: A few years ago, there was a particular case in which compile failed without a trailing newline. We fixed it so that it would work anyway. Unless we are willing for a conforming Python interpreter to fail >>> exec('print("hello")') hello The Reference Manual

[issue33766] Grammar Incongruence

Guido van Rossum added the comment: I am fine with adding this to the docs. But the irony of the case is that the echo command adds a newline, so the original premise (that test.py contains an invalid program) is incorrect. ;-) -- ___ Python track

[issue33766] Grammar Incongruence

Ammar Askar added the comment: Sorry, I was already working on the patch by the time you posted the comment. If we see above, it seems like the tokenize module doesn't correctly mirror the behavior of the C tokenizer. Do you want to try fixing that as a bug? That would involve making a new b

[issue33766] Grammar Incongruence

Change by Ammar Askar : -- keywords: +patch pull_requests: +7009 stage: -> patch review ___ Python tracker ___ ___ Python-bugs-list

[issue33766] Grammar Incongruence

Isaac Elliott added the comment: Cool, thanks for the help. Should I submit a PR with the updated documentation? -- ___ Python tracker ___

[issue33766] Grammar Incongruence

Serhiy Storchaka added the comment: Good point Ammar. Seems there is also a missing corner case in the definition of a physical line: https://docs.python.org/3.8/reference/lexical_analysis.html#physical-lines """ A physical line is a sequence of characters terminated by an end-of-line sequen

[issue33766] Grammar Incongruence

Ammar Askar added the comment: Relevant bit of the parser that emits a fake newline at the end of the file if not present: https://github.com/python/cpython/blob/master/Parser/tokenizer.c#L1059-L1069 -- ___ Python tracker

[issue33766] Grammar Incongruence

Ammar Askar added the comment: Actually, echo implicitly puts a newline at the end. If you run with echo -n, this is the output: $ echo -n 'print("a");print("b")' | python3 -m tokenize 1,0-1,5:NAME 'print' 1,5-1,6:OP '(' 1,6-1,9:STRING

[issue33766] Grammar Incongruence

Isaac Elliott added the comment: I went through that document before I created this issue. I can't find anything which describes this behavior - could you be more specific please? -- ___ Python tracker

[issue33766] Grammar Incongruence

Ammar Askar added the comment: https://docs.python.org/3.8/reference/lexical_analysis.html -- nosy: +ammar2 ___ Python tracker ___

[issue33766] Grammar Incongruence

Isaac Elliott added the comment: Thanks for the clarification. Is there a reference to this in the documentation? -- ___ Python tracker ___ ___

[issue33766] Grammar Incongruence

Serhiy Storchaka added the comment: NEWLINE is not a newline. It is the NEWLINE token. And it is generated at the end of file. $ echo 'print("a");print("b")' | ./python -m tokenize 1,0-1,5:NAME 'print' 1,5-1,6:OP '(' 1,6-1,9:

[issue33766] Grammar Incongruence

New submission from Isaac Elliott : echo 'print("a");print("b")' > test.py This program is grammatically incorrect according to the specification (https://docs.python.org/3.8/reference/grammar.html). But Python 3 runs it without issue. It's this production here simple_stmt: small_stmt (';'