** Also affects: util-linux (Ubuntu Noble)
Importance: Undecided
Status: New
** Also affects: util-linux (Ubuntu Oracular)
Importance: Undecided
Status: New
** Also affects: util-linux (Ubuntu Plucky)
Importance: Undecided
Status: Confirmed
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to util-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2090972
Title:
/boot intermittently fails to mount on boot
Status in util-linux package in Ubuntu:
Confirmed
Status in util-linux source package in Noble:
New
Status in util-linux source package in Oracular:
New
Status in util-linux source package in Plucky:
Confirmed
Bug description:
Starting on Noble, we see /boot fail to mount in approximately one out
of every two thousand boots. The error looks like this:
Found device dev-disk-by\x2dlabel-BOOT.device - QEMU NVMe Ctrl BOOT.
Starting systemd-fsck@dev-disk-by… Check on /dev/disk/by-label/BOOT...
Checking in progress on 1 disk (0.0% complete)
Checking in progress on 0 disks (100.0% complete)
Finished msystemd-fsck@dev-disk-by… Check on /dev/disk/by-label/BOOT.
Mounting boot.mount - /boot...
[ 3.051612] /dev/disk/by-label/BOOT: Can't lookup blockdev
FAILED Failed to mount boot.mount - /boot.
See 'systemctl status boot.mount' for details.
This has resulted in a number of different failure modes for our
users.
Anything that needs to interact with /boot during provisioning will
fail. This is usually something running update-grub or similar.
If we manage to succeed in booting, this can cause subsequent kernel
updates or tools that install kernel modules to fail, because
update-grub fails.
We've _also_ seen this manifest on the root filesystem. In that case,
the boot succeeded, but the by-label links remain absent. When this
occurs, we find installing kernel packages fails because mkinitramfs
can't locate the root disk by label.
It's a vexing problem, and so to reproduce we ran cloud images in a boot
loop until we could reliably reproduce the problem. Unfortunately, we
weren't able to work out anything that made this happen faster, so it's
been a bit slow coming.
It turns out the problem here is that libblkid recently added support to
compute the checksum of the superblocks on ext4 filesystems, and Noble
is the first release to include a version of util-linux new enough to
have this feature. When libblkid determines an ext4 superblock's
checksum is corrupt, it refuses to identify the device as having a
filesystem, which leads to the removeal of the uuid and by-label fields.
systemd-udevd then removes these symlinks. This is where it all goes
wrong.
From our debug traces, it's possible to see this clearly:
(udev-worker)[208]: nvme0n1p16: Probe /dev/nvme0n1p16 with raid and
offset=0
systemd-udevd[208]: 208: libblkid: LOWPROBE: [36] ext4dev:
systemd-udevd[208]: 208: libblkid: BUFFER: reuse: off=1024 len=1024
(for off=1024 len=1024)
systemd-udevd[208]: 208: libblkid: LOWPROBE: magic sboff=56, kboff=1
systemd-udevd[208]: 208: libblkid: LOWPROBE: call probefunc()
systemd-udevd[208]: 208: libblkid: BUFFER: reuse: off=1024 len=1024
(for off=1024 len=1024)
systemd-udevd[208]: incorrect checksum for type ext4dev, got
D919EB5600000000, expected A47F6CF000000000
systemd-udevd[208]: 208: libblkid: LOWPROBE: [37] ext4:
systemd-udevd[208]: 208: libblkid: BUFFER: reuse: off=1024 len=1024
(for off=1024 len=1024)
systemd-udevd[208]: 208: libblkid: LOWPROBE: magic sboff=56, kboff=1
systemd-udevd[208]: 208: libblkid: LOWPROBE: call probefunc()
systemd-udevd[208]: 208: libblkid: BUFFER: reuse: off=1024 len=1024
(for off=1024 len=1024)
systemd-udevd[208]: incorrect checksum for type ext4, got D919EB5600000000,
expected A47F6CF000000000
(udev-worker)[208]: nvme0n1p16: Removing/updating old device symlink
'/dev/disk/by-uuid/ce209fd3-a875-4607-9089-35b9de605bd0', which is no longer
belonging to this device.
(udev-worker)[208]: nvme0n1p16: No reference left for
'/dev/disk/by-uuid/ce209fd3-a875-4607-9089-35b9de605bd0', removing
(udev-worker)[208]: nvme0n1p16: Removing/updating old device symlink
'/dev/disk/by-label/BOOT', which is no longer belonging to this device.
(udev-worker)[208]: nvme0n1p16: No reference left for
'/dev/disk/by-label/BOOT', removing
(udev-worker)[208]: nvme0n1p16: Successfully created symlink
'/dev/block/259:4' to '/dev/nvme0n1p16'
(udev-worker)[208]: nvme0n1p16: sd-device: Created db file
'/run/udev/data/b259:4' for
'/devices/pci0000:00/0000:00:01.0/nvme/nvme0/nvme0n1/nvme0n1p16'
(udev-worker)[208]: nvme0n1p16: Adding watch on '/dev/nvme0n1p16'
(udev-worker)[208]: nvme0n1p16: Device processed (SEQNUM=1630, ACTION=change)
(udev-worker)[208]: nvme0n1p16: sd-device-monitor(worker): Passed 1315 byte
to netlink monitor.
Fortunately, the fix here is straight-forward and is similar to what we
did for resize2fs: use O_DIRECT when reading the superblock. We've
already sent a patch upstream and gotten it accepted there:
https://github.com/util-linux/util-
linux/commit/483c9f38e377ff0b009f546a2c4ee91a1d61588c
We've also been running with a version of this patch backported to
2.39.3-9ubuntu6.1 in our production environment for a couple of months.
It's completely eliminated this problem for us. We were observing this
about once an hour and have had no recurrence since rolling out the fix.
We had a discussion with Ted Ts'o about this as well, and he had some
ideas for future improvements, but nothing that we're implementing in
this fix:
https://lore.kernel.org/util-
linux/6d16e6d83ab48d2ea4402db17c9c0ed5514933a7.1731961869.git.k...@templeofstupid.com/T/#m55eb5087639dcfcfd5708144b1b48caf0cf762b8
I'm attaching the patch that we've applied to our tree and would be
grateful if you'd pull this into util-linux >= 2.39.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/2090972/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
