** No longer affects: mutter (Ubuntu)
** No longer affects: mutter (Ubuntu Focal)
** Changed in: glib2.0 (Ubuntu)
Status: New => Fix Released
** Changed in: glib2.0 (Ubuntu Focal)
Status: New => In Progress
** Changed in: glib2.0 (Ubuntu Focal)
Importance: Undecided => High
** Summary changed:
- gdm fails to start in a VMware Horizon VDI environment with latest mutter
3.36.9-0ubuntu0.20.04.1 in focal-updates
+ glib2.0: Uninitialised memory is written to gschema.compiled, failure to
parse this file leads to gdm, gnome-shell failing to start
** Description changed:
[Impact]
- gdm fails to start in a VMware Horizon VDI environment, with Nvidia GRID
- gpus passed into the VDIs.
+ A recent SRU of mutter 3.36.9-0ubuntu0.20.04.1 caused an outage for a
+ user with 300 VDIs running Focal, where GNOME applications would fail to
+ start, and if you reboot, gdm and gnome-shell both fail to start, and
+ you are left with a black screen and a blinking cursor.
- Downgrading mutter from 3.36.9-0ubuntu0.20.04.1 to 3.36.1-3ubuntu3 in
- -release fixes the issue, and the issue does not occur with
- 3.36.7+git20201123-0.20.04.1.
+ After much investigation, mutter was not at fault. Instead, mutter-
+ common calls the libglib2.0-0 hook on upgrade:
- Currently looking into what landed in bug 1919143 and bug 1905825.
+ Processing triggers for libglib2.0-0:amd64 (2.64.6-1~ubuntu20.04.3) ...
+
+ This in turn calls glib-compile-schemas to recompile the gsettings
+ gschema cache, from the files in /usr/share/glib-2.0/schemas/. The
+ result is a binary gschemas.compiled file, which is loaded by libglib2.0
+ on every invocation of a GNOME application, or gdm or gnome-shell to
+ fetch application default settings.
+
+ Now, glib2.0 2.64.6-1~ubuntu20.04.3 in Focal has some non-deterministic
+ behaviour when calling glib-compile-schemas, causing generated
+ gschemas.compiled files to have differing contents on each run:
+
+ # glib-compile-schemas /usr/share/glib-2.0/schemas
+ # cmp -l /home/ubuntu/schemas/gschemas.compiled
/usr/share/glib-2.0/schemas/gschemas.compiled | gawk '{printf "%08X %02X
%02X\n", $1, strtonum(0$2), strtonum(0$3)}'
+ 0000376F E3 D0
+ 00003771 A4 DB
+
+ # glib-compile-schemas /usr/share/glib-2.0/schemas
+ # cmp -l /home/ubuntu/schemas/gschemas.compiled
/usr/share/glib-2.0/schemas/gschemas.compiled | gawk '{printf "%08X %02X
%02X\n", $1, strtonum(0$2), strtonum(0$3)}'
+ 0000376F E3 C3
+ 00003771 A4 98
+
+ # glib-compile-schemas /usr/share/glib-2.0/schemas
+ # cmp -l /home/ubuntu/schemas/gschemas.compiled
/usr/share/glib-2.0/schemas/gschemas.compiled | gawk '{printf "%08X %02X
%02X\n", $1, strtonum(0$2), strtonum(0$3)}'
+ 0000376F E3 68
+ 00003771 A4 30
+ 00003772 55 56
+
+ The bytes on the left are from a corrupted gschemas.compiled provided by
+ an affected user. The changing bytes on the right are non-deterministic.
+
+ I ran valgrind over glib-compile-schemas, and found that we are writing
+ to uninitialised memory.
+
+ https://paste.ubuntu.com/p/hvZccwdzxz/
+
+ What is happening is that a submodule of glib, gvdb, contains the logic
+ for serialising the gschema data structures, and when it allocates a
+ buffer to store the eventual gschemas.compiled file, it does not
+ initialise it.
+
+ When we populate the fields in the buffer, some bytes are never
+ overwritten, and these junk bytes find themselves written to
+ gschemas.compiled.
+
+ On boot, when gdm and gnome-shell attempt to parse and load this
+ corrupted gschemas.compiled file, it can't parse the junk bytes, and
+ raises and error, which propagates up to a breakpoint in glib logging,
+ but no debugger is present, so the kernel traps the breakpoint, and
+ terminates the library, and the calling application, e.g. gdm.
+
+ The result is that the user is left starting at a black screen with a
+ blinking pointer.
[Testcase]
+ On a Focal system, simply run valgrind over glib-compile-schemas:
+
+ # valgrind glib-compile-schemas /usr/share/glib-2.0/schemas
+
+ You will get output like this, with the warning "Syscall param
+ write(buf) points to uninitialised byte(s)":
+
+ https://paste.ubuntu.com/p/hvZccwdzxz/
+
+ If you happen to have a large amount of gschema overrides present on
+ your system, like my affected user does, you can save a copy of a
+ generated gschema.compiled to your home directory and bindiff it against
+ recompiles:
+
+ # glib-compile-schemas /usr/share/glib-2.0/schemas
+ # cp /usr/share/glib-2.0/schemas/gschema.compiled
/home/ubuntu/schemas/gschemas.compiled
+ # glib-compile-schemas /usr/share/glib-2.0/schemas
+ # cmp -l /home/ubuntu/schemas/gschemas.compiled
/usr/share/glib-2.0/schemas/gschemas.compiled | gawk '{printf "%08X %02X
%02X\n", $1, strtonum(0$2), strtonum(0$3)}'
+ 0000376F E3 C3
+ 00003771 A4 98
+
+ If you install the test package from the following ppa:
+
+ https://launchpad.net/~mruffell/+archive/ubuntu/sf311791-test
+
+ When you run valgrind, it will report a clean run with no writing to
+ uninitialised buffers, and all invocations of glib-compile-schemas will
+ be deterministic, and generate the file same with the same sha256 hash
+ every time. The unwritten bytes if you do a bindiff from before and
+ after will be all set to zero.
+
[Where problems can occur]
+
+ I am doubtful that any programs are relying on buggy non-deterministic
+ behaviour from random bytes found in uninitialised memory, so this
+ should be a relatively safe change.
+
+ Since we are updating glib, which all GNOME applications, gdm and gnome-
+ shell link to, if we introduce an error, it could cause these
+ applications to stop working, and at a worse case, see the symptoms this
+ bug is trying to fix, which is a blinking cursor on a blank screen.
+
+ Installing any updates to glib also causes the gsettings gschema cache
+ to be re-generated, and from this bug, we know that libglib seems to
+ trust the gschema.compiled file and doesn't perform much validation, if
+ the user has bad data in their gschema files, it could lead to their
+ systems having issues on next boot.
+
+ If a regression occurs, users should first attempt to re-generate their
+ schemas like so:
+
+ glib-compile-schemas /usr/share/glib-2.0/schemas
+
+ and if that fails, then they should downgrade their libglib2.0-0
+ libglib2.0-bin libglib2.0-data packages.
+
+ [Other info]
+
+ This was fixed by the commit:
+
+ commit ea64c739239faea463f3cb9154a12cc4532ba525
+ Author: Philip Withnall <withn...@endlessm.com>
+ Date: Wed Mar 18 09:15:59 2020 +0000
+ Subject: gvdb-builder: Initialise some memory to zero in the bloom filter
+ Link:
https://github.com/GNOME/glib/commit/ea64c739239faea463f3cb9154a12cc4532ba525
+
+ Only Focal needs this patch, Groovy and up are unaffected.
--
You received this bug notification because you are a member of Ubuntu
Desktop Bugs, which is subscribed to mutter in Ubuntu.
https://bugs.launchpad.net/bugs/1930359
Title:
glib2.0: Uninitialised memory is written to gschema.compiled, failure
to parse this file leads to gdm, gnome-shell failing to start
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/glib2.0/+bug/1930359/+subscriptions
--
desktop-bugs mailing list
desktop-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/desktop-bugs
