Hi Brian, we have also experienced this issue during our tests, but it's really hard to reproduce it on our side.
It seems to be a regression in virtio-rng and I've prepared a test kernel, but we still haven't found a way to reproduce it systematically, so I couldn't verify if this test kernel is actually fixing the problem or not. Are you able to reproduce this issue easily? In that case, it'd be great if you could a quick test with this kernel: https://kernel.ubuntu.com/~arighi/lp-1989258/ (I've uploaded just the debs here, but if you prefer a ppa let me know) Thanks in advance! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1989258 Title: multiple kernel oops regarding hung tasks delaying boot Status in Auto Package Testing: New Status in linux package in Ubuntu: Confirmed Bug description: Kinetic amd64 instances in the autopkgtest environment are regularly experiencing a slow boot process due to a hung task timeout which happens multiple times. The oops appears below: Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.485373] INFO: task systemd-udevd:140 blocked for more than 1208 seconds. Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.487120] Not tainted 5.19.0-15-generic #15-Ubuntu Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.490936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.492717] task:systemd-udevd state:D stack: 0 pid: 140 ppid: 137 flags:0x00004006 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.494612] Call Trace: Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.495202] <TASK> Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.495727] __schedule+0x23b/0x5c0 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.496559] ? sched_clock_cpu+0x12/0xf0 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.497482] schedule+0x50/0xc0 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.498237] schedule_preempt_disabled+0x15/0x30 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.499306] __mutex_lock.constprop.0+0x4e7/0x760 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.500409] __mutex_lock_slowpath+0x13/0x20 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.501420] mutex_lock+0x36/0x40 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.502211] add_early_randomness+0x1b/0xc0 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.503178] hwrng_register+0x13e/0x1c0 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.504068] virtrng_scan+0x19/0x30 [virtio_rng] Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.505152] virtio_dev_probe+0x1f9/0x2c0 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.506096] really_probe+0x1d1/0x3a0 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.506958] __driver_probe_device+0x11b/0x190 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.507993] driver_probe_device+0x24/0xd0 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.508955] __driver_attach+0xd8/0x200 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.509859] ? __device_attach_driver+0x120/0x120 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.510952] bus_for_each_dev+0x7c/0xc0 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.511846] driver_attach+0x1e/0x30 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.512692] bus_add_driver+0x178/0x220 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.513597] driver_register+0x8f/0xf0 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.514474] ? 0xffffffffc0282000 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.515257] register_virtio_driver+0x20/0x40 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.516277] virtio_rng_driver_init+0x15/0x1000 [virtio_rng] Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.517602] do_one_initcall+0x4a/0x210 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.518498] ? kmem_cache_alloc_trace+0x181/0x300 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.519587] ? do_init_module+0x27/0x200 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.520516] do_init_module+0x50/0x200 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.521403] load_module+0xb60/0xcb0 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.522243] __do_sys_finit_module+0xbd/0x130 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.523255] ? __do_sys_finit_module+0xbd/0x130 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.524312] __x64_sys_finit_module+0x18/0x20 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.525343] do_syscall_64+0x5b/0x80 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.526183] ? syscall_exit_to_user_mode+0x26/0x50 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.527287] ? __x64_sys_mmap+0x33/0x50 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.528180] ? do_syscall_64+0x67/0x80 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.529064] ? exit_to_user_mode_prepare+0x30/0xa0 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.530182] ? syscall_exit_to_user_mode+0x26/0x50 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.531289] ? __x64_sys_newfstatat+0x1c/0x30 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.532310] ? do_syscall_64+0x67/0x80 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.533185] entry_SYSCALL_64_after_hwframe+0x63/0xcd Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.534354] RIP: 0033:0x7f66accbf9fd Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.535191] RSP: 002b:00007ffc83fb0058 EFLAGS: 00000246 ORIG_RAX: 0000000000000139 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.536912] RAX: ffffffffffffffda RBX: 0000555b12b044d0 RCX: 00007f66accbf9fd Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.538541] RDX: 0000000000000000 RSI: 00007f66ace41458 RDI: 0000000000000005 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.540161] RBP: 00007f66ace41458 R08: 0000000000000000 R09: 0000000000000000 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.541803] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000020000 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.543431] R13: 0000555b12b03e60 R14: 0000000000000000 R15: 0000555b12afff80 Sep 9 22:31:32 bdmurray-test-systemd kernel: [ 1330.545063] </TASK> This first occurred with at around 2022-08-24 with at least kernel version 5.19.0-15-generic. But around 2022-09-07 started becoming much more frequent. I've downgraded the version of systemd to 251.2-2ubuntu2 and that did not cause the oops to go away. ProblemType: Bug DistroRelease: Ubuntu 22.10 Package: linux-image-5.19.0-15-generic 5.19.0-15.15 ProcVersionSignature: Ubuntu 5.19.0-15.15-generic 5.19.0 Uname: Linux 5.19.0-15-generic x86_64 AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Sep 9 22:30 seq crw-rw---- 1 root audio 116, 33 Sep 9 22:30 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.23.0-0ubuntu1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A CasperMD5CheckResult: unknown Date: Sat Sep 10 15:51:33 2022 Ec2AMI: ami-00009dfb Ec2AMIManifest: FIXME Ec2AvailabilityZone: nova Ec2InstanceType: autopkgtest Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub Lsusb-t: /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M MachineType: OpenStack Foundation OpenStack Nova PciMultimedia: ProcEnviron: LC_CTYPE=en_US.UTF-8 TERM=screen.xterm-256color PATH=(custom, no user) LANG=C.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.19.0-15-generic root=UUID=63a35cf8-4d22-4a79-a93d-eb178abc8c07 ro console=ttyS0 RelatedPackageVersions: linux-restricted-modules-5.19.0-15-generic N/A linux-backports-modules-5.19.0-15-generic N/A linux-firmware 20220831.gitd3c92280-0ubuntu1 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 04/01/2014 dmi.bios.release: 0.0 dmi.bios.vendor: SeaBIOS dmi.bios.version: Ubuntu-1.8.2-1ubuntu1+esm1 dmi.chassis.type: 1 dmi.chassis.vendor: QEMU dmi.chassis.version: pc-i440fx-xenial dmi.modalias: dmi:bvnSeaBIOS:bvrUbuntu-1.8.2-1ubuntu1+esm1:bd04/01/2014:br0.0:svnOpenStackFoundation:pnOpenStackNova:pvr13.1.4:cvnQEMU:ct1:cvrpc-i440fx-xenial:sku: dmi.product.family: Virtual Machine dmi.product.name: OpenStack Nova dmi.product.version: 13.1.4 dmi.sys.vendor: OpenStack Foundation To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1989258/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
