New submission from Kevin <kad...@us.ibm.com>:
If a native module fails to load, the dynload code will call
PyUnicode_FromString on the error message to give back to the user. This can
cause a UnicodeDecodeError if the locale is not a UTF-8 locale and the error
message contains non-ASCII code points.
While Linux systems almost always use a UTF-8 locale by default nowadays, AIX
systems typically use non-UTF-8 locales by default. We encountered an issue
where a customer did not have libbz2 installed, causing a load failure when bz2
tried to import _bz2 when running in an Italian locale:
$ LC_ALL=it_IT python3 -c 'import bz2'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/QOpenSys/pkgs/lib/python3.6/bz2.py", line 21, in <module>
from _bz2 import BZ2Compressor, BZ2Decompressor
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 161:
invalid continuation byte
After switching to a UTF-8 locale, the problem goes away:
$ LC_ALL=IT_IT python3 -c 'import bz2'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/QOpenSys/pkgs/lib/python3.6/bz2.py", line 21, in <module>
from _bz2 import BZ2Compressor, BZ2Decompressor
ImportError: 0509-022 Impossibile caricare il modulo
/QOpenSys/pkgs/lib/python3.6/lib-dynload/_bz2.so.
0509-150 Il modulo dipendente libbz2.so non è stato caricato.
0509-022 Impossibile caricare il modulo libbz2.so.
0509-026 Errore di sistema: Un file o una directory nel nome percorso
non esiste.
0509-022 Impossibile caricare il modulo
/QOpenSys/pkgs/lib/python3.6/lib-dynload/_bz2.so.
0509-150 Il modulo dipendente
/QOpenSys/pkgs/lib/python3.6/lib-dynload/_bz2.so non è stato caricato.
While this conceivably affects any Unix-like platform, the only system I can
recreate it on is AIX and IBM i PASE. As far as I can tell, on Linux you will
always get something like "error while loading shared libraries: libbz2.so.1.0:
cannot open shared object file: No such file or directory". Even though there
seems to be some translations in GLIBC, I have been unable to get them to be
used on either Fedora or Ubuntu.
----------
components: Interpreter Core
messages: 377713
nosy: kadler
priority: normal
severity: normal
status: open
title: UnicodeDecodeError during load failure in non-UTF-8 locale
type: behavior
versions: Python 3.10, Python 3.5, Python 3.6, Python 3.7, Python 3.8, Python
3.9
_______________________________________
Python tracker <rep...@bugs.python.org>
<https://bugs.python.org/issue41894>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
