Public bug reported:
Problem
=======
Live system with persistence becomes unstable under warm reboot stress. Failures
are intermittent and symptoms vary — like a "time bomb": the system may appear
to work while initrd contents are silently corrupted.
Observed failure modes (~16 reboot stress test):
- Boot hangs in initramfs, then BusyBox (initramfs) shell (no systemd)
- "init not found"
- "Segmentation fault" displayed and never recovers
- Kernel log: "Initramfs unpacking failed: ZSTD-compressed data is corrupt"
(seen on 4 of 15 boots that produced logs; boot continued on those 4)
- 1 of 16 boots hung completely with no userspace logs
- Sometimes boot completes despite errors; unknown whether runtime state is
fully trustworthy
Primary proven failure (binary analysis on hung boot):
On-disk initrd is correct, but runtime overlay.ko in RAM has a single-byte
corruption that invalidates the kernel module signature. modprobe/insmod fails
with "Key was rejected by service"; persistence overlay cannot mount.
Environment
===========
Hardware: Dell OptiPlex AIO Plus 7410 (0RV5V3), BIOS 1.36.0
Intel Core i9-13900, 32 GB RAM
Boot: UEFI, Secure Boot disabled
Boot media: USB live system (ext4 persistence partition)
Initrd: /current/initrd (~177 MiB)
Cmdline: boot=live persistence persistence-label=persistence
live-media=UUID=<ext4-partition> live-media-path=current forcefsck
Packages
========
Kernel: linux-image-6.17.0-41-generic (6.17.0-41.41~24.04.1-generic)
Modules: linux-modules-6.17.0-41-generic
GRUB: grub-efi-amd64-signed 2.06-2ubuntu14.8
Steps to reproduce
==================
1. Build live initrd with live-boot and persistence (~177 MiB).
2. Confirm overlay.ko is in on-disk initrd:
lsinitramfs /current/initrd | grep overlay.ko
3. Run automated warm reboot stress test (~16+ cycles).
4. Observe intermittent symptoms (hang, init not found, segfault, ZSTD error).
5. On initramfs hang, compare overlay.ko from failed runtime with file extracted
from on-disk initrd via unmkinitramfs.
Expected result
===============
Initrd contents in RAM always match on-disk initrd. Every reboot reaches systemd
reliably with persistence.
Actual result
=============
Intermittent initrd corruption in RAM with varying visible symptoms. On proven
failure case: overlay.ko corrupted; modprobe overlay fails; boot stops in
initramfs. On other cycles: system may still boot after errors, leaving unknown
runtime state.
Reboot stress statistics (our test)
====================================
~16 warm reboot cycles total
4 / 15 logged boots: kernel logged "Initramfs unpacking failed:
ZSTD-compressed
data is corrupt" (~1.09s) but boot continued
1 / 16 boots: hung completely, no userspace logs
1 proven failure: overlay.ko RAM vs disk binary mismatch (see below)
(So corruption manifests as warnings on some boots and fatal failure on
others.)
Boot state on proven overlay failure
====================================
- External initrd loads: Freeing initrd memory ~181204K, /init runs
- ext4 live medium mounted; /run/live/medium/rw present
- Manual squashfs mount succeeds
- modprobe overlay → exit 1
- insmod overlay.ko → "Key was rejected by service"
- overlay not in /proc/filesystems; boot stops before systemd
Binary evidence (attachments)
=============================
Two overlay.ko files attached (484313 bytes each):
overlay.ko.from-disk-initrd (from unmkinitramfs of on-disk initrd)
MD5: cec41b97f29dbd6b1b160f7d209b9c68
SHA256: d69e6a2794c0439179364317ca91ab5f5a41fbf0727a27b03139f6645d6c47a6
overlay.ko.from-runtime-failed (from failed initramfs shell)
MD5: 05ea0944dcf1752941fcc35e7780d219
SHA256: 94b827b1c8a7e50800bd206de4ac6f2b79b7d900b40d22e957c549ad2eb048c4
cmp -l: exactly ONE byte differs:
Offset: 194603 (0x2F82B)
Disk byte: 0x01
RAM byte: 0x00
Hex context:
GOOD: ... 00 00 00 00 00 00 0a da 87 02 00 00 00 00 00 [01] 00 00 0d b5 ...
BAD: ... 00 00 00 00 00 00 0a da 87 02 00 00 00 00 00 [00] 00 00 0d b5 ...
ELF section of corrupted byte: .BTF
ELF header: identical
Module signature trailer ("~Module signature appended~"): identical
All other 484312 bytes: identical
Reproduce: cmp -l overlay.ko.from-disk-initrd overlay.ko.from-runtime-failed
→ 194603 1 0
Root cause
==========
Intermittent corruption of initrd contents in RAM after GRUB load. Severity
varies: silent/single-bit flip (overlay.ko signature failure), kernel ZSTD error
message, or other fatal initramfs symptoms. System reliability cannot be
guaranteed even when boot appears to succeed after an error.
Suggested investigation
=======================
- Initrd RAM integrity between GRUB load and kernel cpio unpack on
6.17.0-41-generic
- Relationship between ZSTD unpack errors and single-byte file corruption
- CONFIG_OVERLAY_FS=y as mitigation for live+persistence images
Attachments
===========
1. overlay.ko.from-disk-initrd
2. overlay.ko.from-runtime-failed
** Affects: grub-efi-amd64-signed (Ubuntu)
Importance: Undecided
Status: New
** Attachment added: "Binary evidence: overlay.ko from on-disk initrd (good) vs
failed initramfs runtime (corrupted). Same size (484313 bytes); MD5 differs;
cmp shows 1 byte flip at offset 194603 (0x01→0x00). Includes comparison.txt
with hashes."
https://bugs.launchpad.net/bugs/2162765/+attachment/5989216/+files/overlay-ko-evidence.tar.gz
** Package changed: grub2-unsigned (Ubuntu) => grub-efi-amd64-signed
(Ubuntu)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2162765
Title:
initrd RAM corruption: single-byte flip in overlay.ko invalidates
module signature; intermittent initramfs failures under reboot stress
(6.17.0-41-generic)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/grub-efi-amd64-signed/+bug/2162765/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
