Bastian Blank wrote: >On Wed, Mar 12, 2025 at 04:22:09PM +0000, Steve McIntyre wrote: >> 20241004-1890  works >> 20241110-1927  fails > >Between those two versions is one point release: 12.8. There are no >relevant image build changes. I don't see any relevant package changes. > >> We're struggling to see what might have changed to break things >> here. Could anybody advise please? > >You should get more output on the serial console, most reletant possible >errors from the initramfs. All the cloud images uses the virtual >terminal only as secondary console, with the serial the primary. > >See https://www.suse.com/support/kb/doc/?id=000018805 for example on how >to use serial on vmware.
Nod. Hilariously, with serial attached things run just fine. It looks like it's an issue with stdout/stderr output from the initramfs. Further debugging suggests it's caused by growroot, similar to what was reported a long time ago in https://bugs.launchpad.net/ubuntu/+source/cloud-initramfs-tools/+bug/1123220 With serial output we get: ... Begin: Running /scripts/local-premount ... done. Warning: fsck not present, so skipping root file system [ 2.057922] EXT4-fs (sda1): mounted filesystem with ordered data mode. Quota mode: none. done. Begin: Running /scripts/local-bottom ... [ 2.077695] EXT4-fs (sda1): unmounting filesystem. [ 2.378948] sda: sda1 sda14 sda15 GROWROOT: CHANGED: partition=1 start=262144 old: size=6027264 end=6289407 new: size=524025823 end=524287966 [ 2.455049] EXT4-fs (sda1): mounted filesystem with ordered data mode. Quota mode: none. done. Begin: Running /scripts/init-bottom ... done. ... Without serial output we see (transcribed from one of the images we captured): ... [ 1.390219] EXT4-fs (sda1): mounted filesystem with ordered data mode. Quota mode: none. [ 1.409506] EXT4-fs (sda1): unmounting filesystem. [ 1.715570] sda: sda1 sda14 sda15 [ 1.775936] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000200 ... For now we're just going to make sure that we always turn on the virtual serial console for these VMs, but this does look like a real bug here. Checking dates, it looks like this broke when the latest version of cloud-initramfs-tools (0.18.debian13+deb12u1) was migrated into bookworm with the fix for #1037914. Argh. Most people are not seeing this as the clouds tent to include a serial console... -- Steve McIntyre, Cambridge, UK. st...@einval.com Can't keep my eyes from the circling sky, Tongue-tied & twisted, Just an earth-bound misfit, I...
