Boot-looping or hanging Debian 11 FAI UEFI PXE
Hello, I am wanting to test a deployment for a Debian 11 KVM VM using a Debian 12 KVM VM, but it crashes and bootloops with seemingly no reasoning. - Debian 12 FAI was booting correctly on the same VM - Switching nfsroot config from Bookworm to Bullseye made it bootloop - Replacing `rootovl` by `rd.live.overlay.overlayfs=1 rd.live.overlay.readonly=1`[1] did not help - Remaking the NFSROOT doesn't help - faiserver and fai are both in `NFSROOT_EXTRA_HOSTS` and were correctly contacted in Bookworm The crash seems to come right after NFS mounts but even debug log seem to say nothing, and sometimes it just straight up hangs at IPv6 configuration. Hanging log using `systemd.log_level=debug systemd.log_target=console systemd.journald.forward_to_console=1` : [2] Crash log using `systemd.log_level=debug systemd.log_target=console systemd.journald.forward_to_console=1` : [3] pxelinux.cfg config (tried with a fresh `fai-chboot - : ``` append initrd=initrd.img-5.10.0-29-amd64 ip=dhcp root=/srv/fai/nfsroot:vers=3 rd.live.overlay.overlayfs=1 rd.live.overlay.readonly=1 console=tty0 console=ttyS0,115200 FAI_FLAGS=debug,sshd,createvt FAI_CONFIG_SRC=nfs://faiserver/srv/fai/fai_config/ FAI_ACTION=install systemd.log_level=debug systemd.log_target=console systemd.journald.forward_to_console=1 ``` (nfs://faiserver/srv/fai/fai_config/ is correct and is a NFS share) Are there [1]: https://github.com/dracutdevs/dracut/issues/1565 [2]: Wasn't able to capture that [3]: https://paste.opensuse.org/pastes/28fcc183cacb
Re: Boot-looping or hanging Debian 11 FAI UEFI PXE
Hi Nicolas, > The crash seems to come right after NFS mounts but even debug log seem to say > nothing, and sometimes it just straight up hangs at IPv6 configuration. > > pxelinux.cfg config (tried with a fresh `fai-chboot - : > ``` > append initrd=initrd.img-5.10.0-29-amd64 ip=dhcp > root=/srv/fai/nfsroot:vers=3 rd.live.overlay.overlayfs=1 > rd.live.overlay.readonly=1 console=tty0 console=ttyS0,115200 > FAI_FLAGS=debug,sshd,createvt > FAI_CONFIG_SRC=nfs://faiserver/srv/fai/fai_config/ FAI_ACTION=install > systemd.log_level=debug systemd.log_target=console > systemd.journald.forward_to_console=1 > ``` shouldn't there be something like an NFS server in the configuration? E.g. root=nfs::/srv/fai/nfsroot:vers=3 Or how should it find the the NFS server? Best regards Dirk -- +--+ | Dr. Dirk Geschke / Plankensteinweg 61/ 85435 Erding| | Telefon: 08122-559448 / Mobil: 0176-96906350 / Fax: 08122-9818106 | | d...@geschke-online.de / d...@lug-erding.de / kont...@lug-erding.de | +--+
Re: Boot-looping or hanging Debian 11 FAI UEFI PXE
> On Tue, 7 May 2024 08:13:42 +, Nicolas Formichella > said: > Hello, > I am wanting to test a deployment for a Debian 11 KVM VM using a Debian 12 KVM VM, but it crashes and bootloops with seemingly no reasoning. > ``` > append initrd=initrd.img-5.10.0-29-amd64 ip=dhcp root=/srv/fai/nfsroot:vers=3 rd.live.overlay.overlayfs=1 rd.live.overlay.readonly=1 console=tty0 console=ttyS0,115200 FAI_FLAGS=debug,sshd,createvt FAI_CONFIG_SRC=nfs://faiserver/srv/fai/fai_config/ FAI_ACTION=install systemd.log_level=debug systemd.log_target=console systemd.journald.forward_to_console=1 > ``` FAI never used these options: rd.live.overlay.overlayfs=1 rd.live.overlay.readonly=1 I'm pretty sure they will break things. Instead you have to use the option rootovl. There may be problems if your nfsroot is not build properly. First, which FAI version are you using? Do you use systemd inside the nfsroot? FAI 6.2 uses systemd during the installation and if you still have an older /etc/fai/NFSROOT config file systsemd may be missing inside the nfsroot. If you still have this line in NFSROOT (when using FAI 6.2), please remove it abd rebuild your nfsroot. sysvinit-core systemd-sysv- If you are using FAI < 6.2, the nfsroot does not use systemd and I wonder why you have added some systemd.log... options to the kernel command line. -- regards Thomas
Re: Boot-looping or hanging Debian 11 FAI UEFI PXE
Hello Thomas and thanks all for your answers I am using FAI v.6.2.2 on a Debian 12 base >Do you use systemd inside the nfsroot? Yes > FAI never used these options: rd.live.overlay.overlayfs=1 rd.live.overlay.readonly=1 I'm pretty sure they will break things. Instead you have to use the option rootovl. : This was a debugging attempt, resetting it to ``` append initrd=initrd.img-5.10.0-29-amd64 ip=dhcp root=/srv/fai/nfsroot:vers=3 rootovl FAI_FLAGS=verbose,sshd,createvt FAI_CONFIG_SRC=nfs://faiserver/srv/fai/fai_config/ FAI_ACTION=install ``` Did not fix anything and exhibits the same crashing behaviour, which did not occur when running > If you still have this line in NFSROOT (when using FAI 6.2), please remove it abd rebuild your nfsroot. > sysvinit-core systemd-sysv- This VM was clean-installed with 6.2.2 and does not contain the line in the /etc/fai/NFSROOT file Regards, Nicolas From: linux-fai on behalf of Thomas Lange Sent: Tuesday 7 May 2024 11:09 To: fully automatic installation for Linux Subject: Re: Boot-looping or hanging Debian 11 FAI UEFI PXE > On Tue, 7 May 2024 08:13:42 +, Nicolas Formichella > said: > Hello, > I am wanting to test a deployment for a Debian 11 KVM VM using a Debian 12 KVM VM, but it crashes and bootloops with seemingly no reasoning. > ``` > append initrd=initrd.img-5.10.0-29-amd64 ip=dhcp root=/srv/fai/nfsroot:vers=3 rd.live.overlay.overlayfs=1 rd.live.overlay.readonly=1 console=tty0 console=ttyS0,115200 FAI_FLAGS=debug,sshd,createvt FAI_CONFIG_SRC=nfs://faiserver/srv/fai/fai_config/ FAI_ACTION=install systemd.log_level=debug systemd.log_target=console systemd.journald.forward_to_console=1 > ``` FAI never used these options: rd.live.overlay.overlayfs=1 rd.live.overlay.readonly=1 I'm pretty sure they will break things. Instead you have to use the option rootovl. There may be problems if your nfsroot is not build properly. First, which FAI version are you using? Do you use systemd inside the nfsroot? FAI 6.2 uses systemd during the installation and if you still have an older /etc/fai/NFSROOT config file systsemd may be missing inside the nfsroot. If you still have this line in NFSROOT (when using FAI 6.2), please remove it abd rebuild your nfsroot. sysvinit-core systemd-sysv- If you are using FAI < 6.2, the nfsroot does not use systemd and I wonder why you have added some systemd.log... options to the kernel command line. -- regards Thomas
Re: Boot-looping or hanging Debian 11 FAI UEFI PXE
> On Tue, 7 May 2024 09:59:35 +, Nicolas Formichella > said: > This VM was clean-installed with 6.2.2 and does not contain the line in the /etc/fai/NFSROOT file Please put the log of fai-make-nfsroot -fv (it's also in /var/log/fai/) onto paste.debian.net so I can check if the nfsroot was build properly. regards Thomas
Re: Boot-looping or hanging Debian 11 FAI UEFI PXE
Hello Thomas, `fai-make-nfsroot -fv` : https://paste.debian.net/hidden/5f60f92a/ Regards, Nicolas From: linux-fai on behalf of Thomas Lange Sent: Tuesday 7 May 2024 12:49 To: fully automatic installation for Linux Subject: Re: Boot-looping or hanging Debian 11 FAI UEFI PXE > On Tue, 7 May 2024 09:59:35 +, Nicolas Formichella > said: > This VM was clean-installed with 6.2.2 and does not contain the line in the /etc/fai/NFSROOT file Please put the log of fai-make-nfsroot -fv (it's also in /var/log/fai/) onto paste.debian.net so I can check if the nfsroot was build properly. regards Thomas
Re: Boot-looping or hanging Debian 11 FAI UEFI PXE
> On Tue, 7 May 2024 12:50:34 +, Nicolas Formichella > said: > Hello Thomas, > `fai-make-nfsroot -fv` : https://paste.debian.net/hidden/5f60f92a/ I wonder why your log does not end like mine. This is mine: dracut: Generating /boot/initrd.img-5.10.0-9-amd64 dracut: dracut-squash only supports systemd bases initramfs /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 35: warning: command substitution: ignored null byte in input /usr/lib/dracut/modules.d/45url-lib/module-setup.sh: line 35: warning: command substitution: ignored null byte in input '/srv/fai/nfsroot/boot/vmlinuz-5.10.0-9-amd64' -> '/srv/tftp/fai/vmlinuz-5.10.0-9-amd64' '/srv/fai/nfsroot/boot/initrd.img-5.10.0-9-amd64' -> '/srv/tftp/fai/initrd.img-5.10.0-9-amd64' TFTP environment prepared. Enable DHCP and start the TFTP daemon on root /srv/tftp/fai. FAI packages and related packages inside the nfsroot: dracut 051-1 dracut-live051-1 dracut-network 051-1 dracut-squash 051-1 fai-client 5.10.3 fai-nfsroot5.10.3 fai-setup-storage 5.10.3 Waiting for background jobs to finish [2]+ Running nice xz -q $NFSROOT/var/tmp/base.tar & (wd: /srv/fai/nfsroot) fai-make-nfsroot finished properly. -- regards Thomas
