[issue25051] 'compile' refuses BOM.

2015-09-10 Thread eryksun
eryksun added the comment: You're passing an already decoded string, so the BOM is treated as text. Instead open the file in binary mode, i.e. open("bom3.py", "rb"). This way the BOM will be detected when decoding the source bytes. Here's an example that passes the source as a bytes object:

[issue25051] 'compile' refuses BOM.

2015-09-10 Thread Piotr
Piotr added the comment: Ok I have tried it(I had to remove ЉЊЈЁЂ from *.txt - my cp is 1250): c:\tmp>type test.py compile(open("tokenize_tests-utf8-coding-cookie-and-utf8-bom-sig.txt").read(), "bom3.py", 'exec') c:\tmp>c:\Python34\python.exe test.py Traceback (most recent call last): File "

[issue25051] 'compile' refuses BOM.

2015-09-10 Thread STINNER Victor
STINNER Victor added the comment: Lib/test/test_tokenize.py contains a test for a Python script encoded to UTF-8 and starting with the UTF-8 BOM, Lib/test/tokenize_tests-utf8-coding-cookie-and-utf8-bom-sig.txt. -- ___ Python tracker

[issue25051] 'compile' refuses BOM.

2015-09-10 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: open() uses locale encoding by default. You should specify correct encoding explicitly for portability and unambiguity. >>> compile(open("bom3.py", encoding="utf-8-sig").read(), "bom3.py", 'exec') at 0xb707a7a0, file "bom3.py", line 2> -- nosy: +ser

[issue25051] 'compile' refuses BOM.

2015-09-10 Thread Serhiy Storchaka
Serhiy Storchaka added the comment: open() uses locale encoding by default. You should specify correct encoding explicitly for portability and unambiguity. >>> compile(open("bom3.py", encoding="utf-8-sig").read(), "bom3.py", 'exec') at 0xb707a7a0, file "bom3.py", line 2> -- nosy: +ser

[issue25051] 'compile' refuses BOM.

2015-09-10 Thread Serhiy Storchaka
Changes by Serhiy Storchaka https://t.co/JxJCL5FSwT>: -- nosy: +serhiy.storchaka ___ Python tracker ___ ___ Python-bugs-list mailing l

[issue25051] 'compile' refuses BOM.

2015-09-10 Thread Piotr
New submission from Piotr: Similar to Issue 679880 >>> compile(open("bom3.py").read(), "bom3.py", 'exec') Traceback (most recent call last): File "", line 1, in File "bom3.py", line 1 # coding: utf-8 ^ SyntaxError: invalid character in identifier -- components: Unicode