"Partially revert"? Which part isn't reverted?
Matthias Maier <tam...@43-1.org> writes:
> On Fri, Jun 15, 2018, at 04:42 CDT, Daniel P. Berrangé <berra...@redhat.com>
> wrote:
>
>> On Thu, Jun 14, 2018 at 11:40:41PM -0500, Matthias Maier wrote:
>>> This commit removes the PYTHON_UTF8 workaround. The problem with setting
>>>
>>> LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8
>>>
>>> is that the en_US.UTF-8 locale might not be available. In this case
The workaround is from
commit d4e5ec877ca698a87dabe68814c6f93668f50c60
Author: Daniel P. Berrange <berra...@redhat.com>
Date: Tue Jan 16 13:42:11 2018 +0000
qapi: force a UTF-8 locale for running Python
Python2 did not validate locale correctness when reading input data, so
would happily read UTF-8 data in non-UTF-8 locales. Python3 is strict so
if you try to read UTF-8 data in the C locale, it will raise an error
for any UTF-8 bytes that aren't representable in 7-bit ascii encoding.
e.g.
More background on this can be seen in
https://www.python.org/dev/peps/pep-0538/
Many distros support a new C.UTF-8 locale that is like the C locale,
but with UTF-8 instead of 7-bit ASCII. That is not entirely portable
though. This patch thus sets the LANG to "C", but overrides LC_CTYPE
to be en_US.UTF-8 locale. This gets us pretty close to C.UTF-8, but
in a way that should be portable to everywhere QEMU builds.
This patch only forces UTF-8 for QAPI scripts, since that is the one
showing the immediate error under Python3 with C locale, but potentially
we ought to force this for all python scripts used in the build process.
It's still used only for running QAPI generators.
As far as I can tell, the only non-ASCII input characters are:
* qapi/trace.json
# Copyright (C) 2011-2016 Lluís Vilanova <vilan...@ac.upc.edu>
* tests/qapi-schema/escape-too-big.json
# { 'command': 'é' }
* tests/qapi-schema/unicode-str.json
{ 'command': 'é' }
I believe these characters made Dan put in the workaround. We could get
rid of them if the fix is too onerous (I haven't really looked, yet).
>> What platform are you using where UTF8 locale is not available ?
>
> For example, neither Debian (and for that matter Ubuntu) nor Gentoo
> guarantee that the en_US.UTF-8 locale is available.
>
> We in particular encounter build problems on Gentoo when users have only
> set very specific, non en_US locales, for example de_DE.UTF-8 (or
> similar).
>
>> Indeed I would ideally like to make the entire of QEMU build with an
>> explicit en_US.UTF-8 or C.UTF-8 locale, to ensure that we get reliably
>> reproducible builds, as locale differences have been known to impact
>> output of many tools not just python.
>
> We face the same problem in Gentoo and usually advice users to set
> LC_ALL=C when submitting bug reports. (It is frustrating that glibc
> upstream doesn't get their act together fixing and merging the current
> C.UTF-8 proposal.)
>
> So what about making the build system more robust (by merging the
> patches, or a variant) and either setting C.UTF-8, or C globally
> (depending on availability)?
"git-grep -Fi .utf-8" coughs up
ui/gtk.c: setlocale(LC_CTYPE, "C.UTF-8");
This runs whenever you pick -display gtk. Errors are ignored, though.
commit 27b224a61f97faabbd20bdf72c0c1a3dbe400cd1
Author: Kevin Wolf <kw...@redhat.com>
Date: Tue Jan 31 11:09:45 2017 +0100
gtk: Hardcode LC_CTYPE as C.utf-8
Commit 2cb5d2a4 removed setlocale() for everything except LC_MESSAGES in
order to avoid unwanted side effects such as using the wrong decimal
separator in generated JSON objects. However, the problem that unsetting
LC_CTYPE caused is that non-ASCII characters are considered
non-printable now and therefore the GTK menus display question marks for
accented letters, Chinese characters etc.
A first attempt to fix this [1] was rejected because even just setting
LC_CTYPE to the user's locale (and thereby modifying the semantics of
the ctype.h functions) could have unwanted effects that we're not aware
of yet.
Recently, however, glibc introduced a new locale "C.utf-8" that just
uses UTF-8 as its charset, but otherwise leaves the semantics alone.
Just setting the right character set is enough for our use case, so we
can just hardcode this one without having to be afraid of nasty side
effects.
Older systems that don't have the new locale will continue displaying
question marks, but this should fix the problem for most users.
[1] https://lists.gnu.org/archive/html/qemu-devel/2015-12/msg03591.html
('Re: gtk: use setlocale() for LC_MESSAGES only')
Signed-off-by: Kevin Wolf <kw...@redhat.com>
Message-id: 20170131100945.8189-1-kw...@redhat.com
[ kraxel: change C.utf-8 to C.UTF-8 ]
Signed-off-by: Gerd Hoffmann <kra...@redhat.com>
Looks like your frustration about upstream glibc is quite stale :)
