[issue32555] Encoding issues with the locale encoding

STINNER Victor added the comment: freebsd$ LC_ALL=fr_FR.ISO8859-1 ./python -c 'import locale, os; locale.setlocale(locale.LC_ALL, ""); print(ascii(os.strerror(2)))' 'Fichier ou r\xe9pertoire inexistant' I ran manually this test on FreeBSD: it pass on Python 3.6, 3.7 and master. I close the i

[issue32555] Encoding issues with the locale encoding

STINNER Victor added the comment: New changeset b92c159efada05b3a5ff9d0dbce3fcb2334631f6 by Victor Stinner in branch '3.6': [3.6] bpo-32555: Fix locale encodings (#5193) https://github.com/python/cpython/commit/b92c159efada05b3a5ff9d0dbce3fcb2334631f6 --

[issue32555] Encoding issues with the locale encoding

Change by STINNER Victor : -- keywords: +patch pull_requests: +5047 stage: -> patch review ___ Python tracker ___ ___ Python-bugs-li

[issue32555] Encoding issues with the locale encoding

STINNER Victor added the comment: Example of bug on FreeBSD 11: haypo@freebsd$ LC_ALL=C ./python -c 'import locale, os; locale.setlocale(locale.LC_ALL, "fr_FR.ISO8859-1"); print(ascii(os.strerror(2)))' 'Fichier ou r\udce9pertoire inexistant' Expected result: haypo@freebsd$ LC_ALL=fr_FR.ISO

[issue32555] Encoding issues with the locale encoding

STINNER Victor added the comment: Another issue: _Py_DecodeUTF8Ex() creates surrogate pairs with 16-bit wchar_t (on Windows), whereas input bytes should be escaped. I'm quite sure that it's a bug. -- ___ Python tracker

[issue32555] Encoding issues with the locale encoding

STINNER Victor added the comment: I'm not sure that locale.bindtextdomain() uses the right encoding neither. I propose the following fix: diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c index 324b694b83..1de17d3620 100644 --- a/Modules/_localemodule.c +++ b/Modules/_localemodule

[issue32555] Encoding issues with the locale encoding

New submission from STINNER Victor : Python 3.6 doesn't use the right encoding in os.strerror(), time.stftime(), locale.localeconv(), time.tzname, etc. on macOS, FreeBSD and other platforms. See my fix locale encodings in bpo-29240: commit 7ed7aead9503102d2ed316175f198104e0cd674c, and test_all