Public bug reported:
Kdump fails to dump vmcore even with workaround suggested
This issue is submitted to
track on Power9 Guest where it uses file type qcow2 disk (virtio-scsi)
Boot Log: (Attached full console log)
[ 3.754031] 32regs : 19616.000 MB/sec
[ 3.794031] 32regs_prefetch: 17280.000 MB/sec
[ 3.834030] altivec : 22480.000 MB/sec
[ 3.834063] xor: using function: altivec (22480.000 MB/sec)
done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
Begin: Waiting for root file system ... Begin: Running /scripts/local-block ...
mdadm: No devices listed in conf file were found.
done.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
done.
Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=5e1fe9e9-cf03-4c73-adce-0e57676f98e0 does not exist. Dropping to
a shell!
BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu4) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
Contact Information = Balamuruhan S / balamuruh...@in.ibm.com
---uname output---
Guest Kernel: 4.18.0-11-generic
Host Kernel: 4.18.0-11-generic
Machine Type = Boston
---Debugger---
A debugger is not configured
---Steps to Reproduce---
1. Have a healthy KVM guest with Ubuntu 1810 with kernel 4.18.0-11-generic
2. Install kdump, kexec and crash tools in the guest,
# dpkg -l | grep crash
ii apport 2.20.10-0ubuntu13.1
all automatically generate crash reports for debugging
ii crash 7.2.3+real-1
ppc64el kernel debugging utility, allowing gdb like syntax
ii kdump-tools 1:1.6.4-2ubuntu1
ppc64el scripts and tools for automating kdump (Linux crash dumps)
ii linux-crashdump 4.18.0.11.12
ppc64el Linux kernel crashdump setup for the latest generic kernel
ii python3-apport 2.20.10-0ubuntu13.1
all Python 3 library for Apport crash report handling
3. Ensure workaround suggested in Bug 172389 is followed by uncomment the
`KDUMP_CMDLINE_APPEND`
and change nr_cpus to maxcpus in /etc/default/kdump-tools config file,
# cat /etc/default/kdump-tools | grep -i cmdline
# KDUMP_CMDLINE - The default is to use the contents of /proc/cmdline.
# Set this variable to override /proc/cmdline.
# KDUMP_CMDLINE_APPEND - Additional arguments to append to the command line
#KDUMP_CMDLINE=""
KDUMP_CMDLINE_APPEND="1 maxcpus=1 systemd.unit=kdump-tools.service irqpoll
noirqdistrib nousb reset_devices"
4. restart the kdump tools service,
# service kdump-tools restart
# service kdump-tools status
? kdump-tools.service - Kernel crash dump capture service
Loaded: loaded (/lib/systemd/system/kdump-tools.service; enabled; vendor pres
Active: active (exited) since Mon 2018-12-03 02:34:03 CST; 1 weeks 3 days ago
Process: 1560 ExecStart=/etc/init.d/kdump-tools start (code=exited, status=0/S
Main PID: 1560 (code=exited, status=0/SUCCESS)
Dec 03 02:34:02 ubuntu1810 systemd[1]: Starting Kernel crash dump capture servic
Dec 03 02:34:02 ubuntu1810 kdump-tools[1560]: Starting kdump-tools: * Creating
Dec 03 02:34:02 ubuntu1810 kdump-tools[1560]: * Creating symlink /var/lib/kdump
Dec 03 02:34:03 ubuntu1810 kdump-tools[1560]: Modified cmdline:BOOT_IMAGE=/boot/
Dec 03 02:34:03 ubuntu1810 kdump-tools[1560]: * loaded kdump kernel
Dec 03 02:34:03 ubuntu1810 kdump-tools[1678]: /sbin/kexec -p --command-line="BOO
Dec 03 02:34:03 ubuntu1810 kdump-tools[1679]: loaded kdump kernel
Dec 03 02:34:03 ubuntu1810 systemd[1]: Started Kernel crash dump capture service
5. check kdump-config is state is ready to dump,
# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr:
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.18.0-11-generic
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to
/var/lib/kdump/initrd.img-4.18.0-11-generic
current state: ready to kdump
6. Reboot the guest and check for kernel cmdline whether crashkernel is
included,
# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinux-4.18.0-11-generic
root=UUID=5e1fe9e9-cf03-4c73-adce-0e57676f98e0 ro net.ifnames=0 biosdevname=0
crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M@128M
7. enable sysrq and trigger the crash,
# echo 1 > /proc/sys/kernel/sysrq
# cat /proc/sys/kernel/sysrq
1
# echo c > /proc/sysrq-trigger
kdump fails to generate vmcore after crash instead reboots and enters
initramfs
Attachment:
1. Guest console log
2. Guest sosreport
3. Guest xml
** Affects: linux (Ubuntu)
Importance: Undecided
Assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
Status: New
** Tags: architecture-ppc64le bugnameltc-174078 severity-high
targetmilestone-inin---
** Tags added: architecture-ppc64le bugnameltc-174078 severity-high
targetmilestone-inin---
** Changed in: ubuntu
Assignee: (unassigned) => Ubuntu on IBM Power Systems Bug Triage
(ubuntu-power-triage)
** Package changed: ubuntu => linux (Ubuntu)
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1808743
Title:
[Ubuntu 1810] Kdump fails to dump vmcore and enters initramfs inside
Power9 KVM guest
Status in linux package in Ubuntu:
New
Bug description:
Kdump fails to dump vmcore even with workaround suggested
This issue is submitted to
track on Power9 Guest where it uses file type qcow2 disk (virtio-scsi)
Boot Log: (Attached full console log)
[ 3.754031] 32regs : 19616.000 MB/sec
[ 3.794031] 32regs_prefetch: 17280.000 MB/sec
[ 3.834030] altivec : 22480.000 MB/sec
[ 3.834063] xor: using function: altivec (22480.000 MB/sec)
done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ...
done.
Begin: Running /scripts/local-premount ... done.
Begin: Waiting for root file system ... Begin: Running /scripts/local-block
... mdadm: No devices listed in conf file were found.
done.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
mdadm: No devices listed in conf file were found.
done.
Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=5e1fe9e9-cf03-4c73-adce-0e57676f98e0 does not exist. Dropping
to a shell!
BusyBox v1.27.2 (Ubuntu 1:1.27.2-2ubuntu4) built-in shell (ash)
Enter 'help' for a list of built-in commands.
(initramfs)
Contact Information = Balamuruhan S / balamuruh...@in.ibm.com
---uname output---
Guest Kernel: 4.18.0-11-generic
Host Kernel: 4.18.0-11-generic
Machine Type = Boston
---Debugger---
A debugger is not configured
---Steps to Reproduce---
1. Have a healthy KVM guest with Ubuntu 1810 with kernel 4.18.0-11-generic
2. Install kdump, kexec and crash tools in the guest,
# dpkg -l | grep crash
ii apport 2.20.10-0ubuntu13.1
all automatically generate crash reports for debugging
ii crash 7.2.3+real-1
ppc64el kernel debugging utility, allowing gdb like syntax
ii kdump-tools 1:1.6.4-2ubuntu1
ppc64el scripts and tools for automating kdump (Linux crash dumps)
ii linux-crashdump 4.18.0.11.12
ppc64el Linux kernel crashdump setup for the latest generic kernel
ii python3-apport 2.20.10-0ubuntu13.1
all Python 3 library for Apport crash report handling
3. Ensure workaround suggested in Bug 172389 is followed by uncomment the
`KDUMP_CMDLINE_APPEND`
and change nr_cpus to maxcpus in /etc/default/kdump-tools config file,
# cat /etc/default/kdump-tools | grep -i cmdline
# KDUMP_CMDLINE - The default is to use the contents of /proc/cmdline.
# Set this variable to override /proc/cmdline.
# KDUMP_CMDLINE_APPEND - Additional arguments to append to the command line
#KDUMP_CMDLINE=""
KDUMP_CMDLINE_APPEND="1 maxcpus=1 systemd.unit=kdump-tools.service irqpoll
noirqdistrib nousb reset_devices"
4. restart the kdump tools service,
# service kdump-tools restart
# service kdump-tools status
? kdump-tools.service - Kernel crash dump capture service
Loaded: loaded (/lib/systemd/system/kdump-tools.service; enabled; vendor
pres
Active: active (exited) since Mon 2018-12-03 02:34:03 CST; 1 weeks 3 days
ago
Process: 1560 ExecStart=/etc/init.d/kdump-tools start (code=exited,
status=0/S
Main PID: 1560 (code=exited, status=0/SUCCESS)
Dec 03 02:34:02 ubuntu1810 systemd[1]: Starting Kernel crash dump capture
servic
Dec 03 02:34:02 ubuntu1810 kdump-tools[1560]: Starting kdump-tools: *
Creating
Dec 03 02:34:02 ubuntu1810 kdump-tools[1560]: * Creating symlink
/var/lib/kdump
Dec 03 02:34:03 ubuntu1810 kdump-tools[1560]: Modified
cmdline:BOOT_IMAGE=/boot/
Dec 03 02:34:03 ubuntu1810 kdump-tools[1560]: * loaded kdump kernel
Dec 03 02:34:03 ubuntu1810 kdump-tools[1678]: /sbin/kexec -p
--command-line="BOO
Dec 03 02:34:03 ubuntu1810 kdump-tools[1679]: loaded kdump kernel
Dec 03 02:34:03 ubuntu1810 systemd[1]: Started Kernel crash dump capture
service
5. check kdump-config is state is ready to dump,
# kdump-config show
DUMP_MODE: kdump
USE_KDUMP: 1
KDUMP_SYSCTL: kernel.panic_on_oops=1
KDUMP_COREDIR: /var/crash
crashkernel addr:
/var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinux-4.18.0-11-generic
kdump initrd:
/var/lib/kdump/initrd.img: symbolic link to
/var/lib/kdump/initrd.img-4.18.0-11-generic
current state: ready to kdump
6. Reboot the guest and check for kernel cmdline whether crashkernel is
included,
# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinux-4.18.0-11-generic
root=UUID=5e1fe9e9-cf03-4c73-adce-0e57676f98e0 ro net.ifnames=0 biosdevname=0
crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M@128M
7. enable sysrq and trigger the crash,
# echo 1 > /proc/sys/kernel/sysrq
# cat /proc/sys/kernel/sysrq
1
# echo c > /proc/sysrq-trigger
kdump fails to generate vmcore after crash instead reboots and enters
initramfs
Attachment:
1. Guest console log
2. Guest sosreport
3. Guest xml
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1808743/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
