I am seeing a lot of hard lockups when I fire up a KVM (Windows 10 with VFIO), on 4.19.2 and 4.19.3 (tested from linux-stable). System is a Xeon E5-2667v2 on Supermicro X9SRA. Kernel command line: consoleblank=0 intel_iommu=on rcu_nocbs=1-15 nohz_full=1-15 transparent_hugepage=never audit=0 pcie_aspm=off Doing a git bisect revealed as the culprit:
commit 9cf57731b63e37ed995b46690adc604891a9a28f Author: Peter Zijlstra <mailto:pet...@infradead.org> Date: Thu Jun 7 10:52:03 2018 +0200 watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work Oleg suggested to replace the "watchdog/%u" threads with cpu_stop_work. That removes one thread per CPU while at the same time fixes softlockup vs SCHED_DEADLINE. But more importantly, it does away with the single smpboot_update_cpumask_percpu_thread() user, which allows cleanups/shrinkage of the smpboot interface. Suggested-by: Oleg Nesterov <mailto:o...@redhat.com> Signed-off-by: Peter Zijlstra (Intel) <mailto:pet...@infradead.org> Cc: Linus Torvalds <mailto:torva...@linux-foundation.org> Cc: Peter Zijlstra <mailto:pet...@infradead.org> Cc: Thomas Gleixner <mailto:t...@linutronix.de> Cc: mailto:linux-kernel@vger.kernel.org Signed-off-by: Ingo Molnar <mailto:mi...@kernel.org> -----------------8<----------------- NMI watchdog: Watchdog detected hard LOCKUP on cpu 10 Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill 8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64 hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801 lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio CPU: 10 PID: 12844 Comm: fixdep Not tainted 4.19.3-9-custom19 #1 Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015 RIP: 0010:__rcu_read_lock+0xe/0x20 Code: df 48 85 d2 40 0f 94 c5 e8 2f 2f 73 00 89 e8 5b 5d c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 65 48 8b 04 25 40 5c 01 00 <83> 80 90 03 00 00 01 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 RSP: 0018:ffff9f0721e67c40 EFLAGS: 00000246 RAX: ffff8e13ef389e40 RBX: ffff8e13ef389e40 RCX: 0000000000000000 RDX: 0000000000000041 RSI: 00000000006080c0 RDI: ffff8e13e2373300 RBP: ffff8e13e2373300 R08: 0000000000000003 R09: 0000000000000000 R10: 0000000000000fe0 R11: 0000000000000000 R12: 00000000006080c0 R13: ffffffffaf4766aa R14: ffff8e1cbf57ea00 R15: ffff8e1cbf57ea00 FS: 00007fc6b422a540(0000) GS:ffff8e1cbfa80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000555bdbfdc5f8 CR3: 0000001732010004 CR4: 00000000001606e0 Call Trace: get_mem_cgroup_from_mm+0x16/0xf0 memcg_kmem_get_cache+0x6a/0x2a0 kmem_cache_alloc+0xf7/0x1c0 __alloc_file+0x2a/0xe0 ? ext4_file_mmap+0xa0/0xa0 [ext4] alloc_empty_file+0x43/0xc0 ? terminate_walk+0x91/0x100 path_openat+0x4c/0x14e0 ? current_time+0x4f/0x90 ? atime_needs_update+0x77/0xe0 ? touch_atime+0x33/0xe0 do_filp_open+0x93/0x100 ? __check_object_size+0x102/0x189 ? _raw_spin_unlock+0x16/0x30 do_sys_open+0x186/0x210 do_syscall_64+0x5b/0x170 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7fc6b41514c2 Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 85 70 0d 00 8b 00 85 c0 75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff 0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25 RSP: 002b:00007fffe98405b0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 RAX: ffffffffffffffda RBX: 0000555bdbfd28b3 RCX: 00007fc6b41514c2 RDX: 0000000000000000 RSI: 0000555bdbfd28b3 RDI: 00000000ffffff9c RBP: 0000555bdbfd28b3 R08: 0000555bdbfd28b3 R09: 00007fffe98401d0 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007fffe9841c54 R14: 0000000000000015 R15: 00007fffe9841c54 NMI watchdog: Watchdog detected hard LOCKUP on cpu 12 Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill 8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64 hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801 lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio CPU: 12 PID: 13040 Comm: cc1 Not tainted 4.19.3-9-custom19 #1 Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015 RIP: 0033:0x6ae07f Code: 85 46 ff ff ff 66 0f 1f 44 00 00 4c 8b 77 18 49 89 fd 4d 85 f6 0f 84 d0 00 00 00 49 8b 06 48 85 c0 0f 84 f4 00 00 00 48 8b 10 <48> 8b 48 08 48 85 d2 0f 84 84 01 00 00 49 89 16 48 89 4a 08 66 0f RSP: 002b:00007fff1afce640 EFLAGS: 00000206 RAX: 00000000025baba0 RBX: 0000000000000000 RCX: 0000000000000007 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000268d210 RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000025f51a0 R10: 0000000000000001 R11: 00000000024f0ca8 R12: 0000000000000080 R13: 000000000268d210 R14: 00000000024874b0 R15: 00000000024cbe40 FS: 00007f8fddc7dac0 GS: 0000000000000000 NMI watchdog: Watchdog detected hard LOCKUP on cpu 6 Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill 8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64 hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801 lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio CPU: 6 PID: 14001 Comm: cc1 Not tainted 4.19.3-9-custom19 #1 Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015 RIP: 0033:0x12b0328 Code: a3 ff ff 48 8b 45 38 48 8b 70 10 8b 03 8d 78 01 eb cf 41 57 41 56 49 89 f6 48 83 c6 14 41 55 41 54 55 53 48 89 fb 48 83 ec 38 <48> 8b af c0 01 00 00 e8 7c ff ff ff 48 89 df 48 89 83 c0 01 00 00 RSP: 002b:00007fffc9ea1ce0 EFLAGS: 00000202 RAX: 0000000002cb67a0 RBX: 0000000002c45590 RCX: 0000000000000001 RDX: 0000000000000035 RSI: 00007f7ed3d61df4 RDI: 0000000002c45590 RBP: 0000000000000000 R08: 000000000000023c R09: 00007f7ed515c000 R10: 000000000dd86d60 R11: 0000000000000000 R12: 00007f7ed3d3c3d8 R13: 0000000000000000 R14: 00007f7ed3d61de0 R15: 00007f7ed3d61de0 FS: 00007f7ed4688ac0 GS: 0000000000000000 NMI watchdog: Watchdog detected hard LOCKUP on cpu 14 Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill 8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64 hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801 lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio CPU: 14 PID: 14175 Comm: cc1 Not tainted 4.19.3-9-custom19 #1 Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015 RIP: 0033:0x7fe6e03faba0 Code: 5d 41 5c 41 5d 41 5e 41 5f c3 49 89 fd e8 48 87 fe ff 49 83 c5 02 45 0f b7 65 fe 49 89 c7 4c 8d 70 fe 48 85 db 75 10 eb cf 90 <48> 83 c5 08 48 8b 5d 00 48 85 db 74 c1 66 44 3b 23 75 ed 48 8d 7b RSP: 002b:00007fffbc0666a0 EFLAGS: 00000283 RAX: 0000000000000008 RBX: 00007fffbc06ea62 RCX: 00000000000003e7 RDX: 0000000000004900 RSI: 0000000002a807c1 RDI: 00007fe6e05453e7 RBP: 00007fffbc06b958 R08: 0000000000000000 R09: 0000000000000000 R10: 000000001a6677ae R11: 0000000000000000 R12: 000000000000414c R13: 00007fe6e05453e9 R14: 0000000000000006 R15: 0000000000000008 FS: 00007fe6e03beac0 GS: 0000000000000000 NMI watchdog: Watchdog detected hard LOCKUP on cpu 2 Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill 8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64 hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801 lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio CPU: 2 PID: 14846 Comm: cc1 Not tainted 4.19.3-9-custom19 #1 Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015 RIP: 0033:0x12aa6d1 Code: e8 64 42 30 ff 48 89 df 48 85 db 75 ec 5b c3 66 2e 0f 1f 84 00 00 00 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 55 <53> 48 89 f3 48 83 ec 08 48 8b 57 40 48 8b 42 10 48 8b 4a 18 48 29 RSP: 002b:00007ffe4da64ca0 EFLAGS: 00000202 RAX: 00000000031b2300 RBX: 0000000000000001 RCX: 0000000000000000 RDX: 00007ffe4da64d3c RSI: 0000000000000002 RDI: 00000000030336a0 RBP: 0000000003033d70 R08: 000000000303a110 R09: 0000000000000000 R10: 000000000303a110 R11: 0000000000000000 R12: 00000000031b23de R13: 0000000001a83040 R14: 00000000030336a0 R15: 00007ffe4da64d3c FS: 00007f78a8eecac0 GS: 0000000000000000 NMI watchdog: Watchdog detected hard LOCKUP on cpu 13 Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill 8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64 hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801 lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio CPU: 13 PID: 15862 Comm: cc1 Not tainted 4.19.3-9-custom19 #1 Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015 RIP: 0033:0x12b71a1 Code: 4c 24 08 e8 41 21 03 00 49 8b 4d 18 4c 8b 4c 24 08 e9 63 ff ff ff 41 80 4d 50 02 eb 92 0f 1f 40 00 39 55 08 0f 85 f5 fc ff ff <48> 8b 7d 00 89 4c 24 20 44 89 5c 24 28 4c 89 44 24 18 48 89 54 24 RSP: 002b:00007ffdd11d70c0 EFLAGS: 00000246 RAX: 00000000000042f0 RBX: 00000000000042f0 RCX: 00000000f630c2f0 RDX: 0000000000000005 RSI: 0000000002beb4d7 RDI: 00000000029e4a70 RBP: 00007fd9671b4108 R08: 0000000002b12c30 R09: 0000000000000005 R10: 0000000000008000 R11: 0000000000008000 R12: 0000000002a43c58 R13: 00000000029e4a70 R14: 00007ffdd11d71bc R15: 0000000000007fff FS: 00007fd967691ac0 GS: 0000000000000000 NMI watchdog: Watchdog detected hard LOCKUP on cpu 11 Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill 8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64 hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801 lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio CPU: 11 PID: 16002 Comm: cc1 Not tainted 4.19.3-9-custom19 #1 Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015 RIP: 0033:0x5e8bae Code: c1 29 c3 89 d8 4d 8d 2c c7 49 8b 45 00 48 85 c0 0f 84 02 01 00 00 89 d5 49 89 f6 49 89 fc 48 83 f8 01 74 4e 48 8b 36 4c 8b 00 <4c> 89 54 24 18 89 4c 24 10 48 63 46 08 48 8b 36 4c 89 c7 44 89 4c RSP: 002b:00007ffd42b14d60 EFLAGS: 00000206 RAX: 000000000143ee48 RBX: 00000000000000ce RCX: 0000000000000007 RDX: 0000000000166e6e RSI: 00007ffd42b14de0 RDI: 0000000002a7e810 RBP: 0000000000166e6e R08: 000000000143d459 R09: 00000000000000fb R10: 0000000000000005 R11: 0000000000166e6e R12: 0000000002a7e810 R13: 0000000002af7e70 R14: 00007ffd42b14dd8 R15: 0000000002af7800 FS: 00007ff5800a6ac0 GS: 0000000000000000 NMI watchdog: Watchdog detected hard LOCKUP on cpu 3 Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill 8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64 hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801 lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio CPU: 3 PID: 16417 Comm: cc1 Not tainted 4.19.3-9-custom19 #1 Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015 RIP: 0033:0xcdba17 Code: 00 00 00 48 89 f2 89 f5 bf 02 00 00 00 48 83 fa 01 0f 87 7d fd ff ff eb 9a 66 2e 0f 1f 84 00 00 00 00 00 48 8b 05 71 c5 13 01 <89> d5 8b b8 ac 01 00 00 8d 47 ff 48 98 48 39 d0 0f 82 54 fd ff ff RSP: 002b:00007ffcd5b770e0 EFLAGS: 00000287 RAX: 0000000002256390 RBX: 00007fa6e85d10a8 RCX: 0000000000000080 RDX: 0000000000000008 RSI: 0000000000000008 RDI: 0000000000000080 RBP: 000000002ca727c4 R08: 0000000000000080 R09: 0000000000000001 R10: 0000000000000008 R11: 0000000000000001 R12: 00007fa6e7d09d20 R13: 0000000000000001 R14: 0000000000000002 R15: 00007fa6e7d0c488 FS: 00007fa6e8a9cac0 GS: 0000000000000000 NMI watchdog: Watchdog detected hard LOCKUP on cpu 15 Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill 8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64 hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801 lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio CPU: 15 PID: 16522 Comm: cc1 Not tainted 4.19.3-9-custom19 #1 Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015 RIP: 0033:0x129dfe0 Code: 84 c0 0f 84 23 02 00 00 48 8b 4d 08 48 8d 15 a7 2f 7c 00 48 89 df 31 c0 be 06 00 00 00 e8 98 24 00 00 e9 04 02 00 00 0f 1f 00 <48> 8b 45 08 0f b6 40 10 a8 01 0f 84 80 00 00 00 d0 e8 0f b6 d0 83 RSP: 002b:00007fff2c18a540 EFLAGS: 00000246 RAX: 0000000000000035 RBX: 00000000029bf6a0 RCX: 0000000000000000 RDX: 0b5a02400b5a01c0 RSI: 0000000000000000 RDI: 0000000000000000 RBP: 00000000029bfc48 R08: 00000000000001b3 R09: 00007f9025d4c000 R10: 000000000b5a01c0 R11: 0000000000000000 R12: 0000000000000000 R13: 0000000000000000 R14: 0000000000000000 R15: 00000000029bf6a0 FS: 00007f902527eac0 GS: 0000000000000000 NMI watchdog: Watchdog detected hard LOCKUP on cpu 7 Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill 8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64 hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801 lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio CPU: 7 PID: 16495 Comm: cc1 Not tainted 4.19.3-9-custom19 #1 Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015 RIP: 0033:0x12b2248 Code: 41 59 e9 63 ff ff ff 0f 1f 00 8b b4 24 a8 00 00 00 c6 84 24 93 00 00 00 00 85 f6 0f 84 49 ff ff ff 90 45 0f b6 9e 85 04 00 00 <3c> 50 75 8c 41 8b 4d 08 41 0f b7 55 06 8d 41 ff 48 8d 04 40 48 c1 RSP: 002b:00007ffff6bead70 EFLAGS: 00000246 RAX: 0000000000000035 RBX: 0000000000000028 RCX: 00000000023f2dc0 RDX: 00007f16bd10a160 RSI: 0000000000000000 RDI: 00007f16bbfdb7f8 RBP: 0000000002440000 R08: 0008000000000000 R09: 00007f16bdbecc10 R10: 0000000002205010 R11: 0000000000000002 R12: 0000000000000000 R13: 00007f16bd4ce090 R14: 00000000022796a0 R15: 0000000000000000 FS: 00007f16bda2bac0 GS: 0000000000000000 NMI watchdog: Watchdog detected hard LOCKUP on cpu 5 Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill 8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64 hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801 lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio CPU: 5 PID: 15214 Comm: cc1 Not tainted 4.19.3-9-custom19 #1 Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015 RIP: 0033:0xa2f900 Code: 0c 48 8b 51 08 49 89 14 ef 48 83 c5 01 48 8b 5c c3 08 48 85 db 75 a8 0f 1f 00 48 85 ed 74 6b 48 83 ed 01 49 8b 1c ef eb 91 90 <4c> 8b 53 08 41 0f b7 02 83 3c 85 40 b2 4a 01 0c 48 89 c1 75 86 83 RSP: 002b:00007fffd880d650 EFLAGS: 00000246 RAX: 0000000000000054 RBX: 00007f91fbe22ae0 RCX: 00007f91fbe22b00 RDX: 0000000000000000 RSI: 0000000000725f70 RDI: 00007f91fbe22af8 RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000078 R10: 0000000000000078 R11: 0000000000000000 R12: 0000000000000001 R13: 00007fffd880d760 R14: 0000000000725f70 R15: 00007fffd880d660 FS: 00007f92050edac0 GS: 0000000000000000 NMI watchdog: Watchdog detected hard LOCKUP on cpu 8 Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill 8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64 hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801 lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio CPU: 8 PID: 19549 Comm: genksyms Not tainted 4.19.3-9-custom19 #1 Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015 RIP: 0033:0x7f00f83a1b50 Code: 15 ad f7 11 00 0f 83 e2 00 00 00 48 39 f7 72 0f 74 12 4c 8d 0c 16 4c 39 cf 0f 82 63 01 00 00 48 89 d1 f3 a4 c3 80 fa 08 73 12 <80> fa 04 73 1e 80 fa 01 77 26 72 05 0f b6 0e 88 0f c3 48 8b 4c 16 RSP: 002b:00007ffdcfd167f8 EFLAGS: 00000297 RAX: 000055b0cd5eef90 RBX: 000055b0cd5eeb40 RCX: 000055b0ccd58010 RDX: 0000000000000002 RSI: 000055b0ccd585f7 RDI: 000055b0cd5eef90 RBP: 000000000000003a R08: 000055b0ccbe5860 R09: 000055b0ccd585f7 R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000002 R13: 0000000000000001 R14: 000055b0ccbe5fc0 R15: 000055b0ccbe61a0 FS: 00007f00f84c3540 GS: 0000000000000000 NMI watchdog: Watchdog detected hard LOCKUP on cpu 9 Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill 8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64 hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801 lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio CPU: 9 PID: 20039 Comm: cc1 Not tainted 4.19.3-9-custom19 #1 Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015 RIP: 0033:0x67b7d2 Code: 83 c4 08 31 c0 5b 5d 41 5c 41 5d c3 0f 1f 40 00 bf 88 95 42 01 31 c0 e8 6c e6 c0 00 eb dc 66 2e 0f 1f 84 00 00 00 00 00 41 57 <41> 56 41 55 41 54 55 53 48 81 ec 88 00 00 00 48 89 0c 24 64 48 8b RSP: 002b:00007ffc28206fb0 EFLAGS: 00000202 RAX: 0000000000000001 RBX: 0000000000000004 RCX: 00007ffc282070c8 RDX: 00007f86a1e84680 RSI: 0000000000000004 RDI: 0000000000000000 RBP: 00007f86a1e84680 R08: 0000000000000000 R09: 00007ffc282070c8 R10: 0000000007571fe8 R11: 0000000000000008 R12: 0000000007571fe8 R13: 00007f86a1f237e0 R14: 00007f86a1f31c80 R15: 00007f86a1e84680 FS: 00007f86a26d6ac0 GS: 0000000000000000 NMI watchdog: Watchdog detected hard LOCKUP on cpu 1 Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill 8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64 hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801 lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg crypto_user ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1 vfio CPU: 1 PID: 20175 Comm: cc1 Not tainted 4.19.3-9-custom19 #1 Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015 RIP: 0033:0x12ae100 Code: 87 c3 0f 1f 84 00 00 00 00 00 48 63 47 28 4c 8b 4f 18 44 8b 47 24 48 89 c6 48 8d 04 80 49 8d 04 c1 39 10 76 45 44 39 c6 73 78 <42> 8d 04 06 d1 e8 89 c1 48 8d 0c 89 41 39 14 c9 76 1a eb 44 0f 1f RSP: 002b:00007ffc4825b420 EFLAGS: 00000297 RAX: 000000000000508f RBX: 000000007ffb0bec RCX: 000000000000503f RDX: 000000007ffb14e4 RSI: 0000000000005040 RDI: 00007fbdd9a4e000 RBP: 000000007ffb0bc2 R08: 000000000000508f R09: 00007fbdd82c7000 R10: 000000007ffb14e4 R11: 00007fbdd9a4e000 R12: 0000000000000000 R13: 0000000000000001 R14: 0000000000000000 R15: 000000007ffb0bea FS: 00007fbdd8f72ac0 GS: 0000000000000000 perf: interrupt took too long (2503 > 2500), lowering kernel.perf_event_max_sample_rate to 79800 logitech-hidpp-device 0003:046D:101B.000A: HID++ 1.0 device connected. perf: interrupt took too long (3139 > 3128), lowering kernel.perf_event_max_sample_rate to 63600 br0: port 2(vethPC49WT) entered disabled state device vethPC49WT left promiscuous mode br0: port 2(vethPC49WT) entered disabled state e1000e: eno2 NIC Link is Down br0: port 1(eno2) entered disabled state device eno2 left promiscuous mode br0: port 1(eno2) entered disabled state systemd-shutdow: 50 output lines suppressed due to ratelimiting shutdown[1]: Syncing filesystems and block devices. shutdown[1]: Sending SIGTERM to remaining processes... -----------------8<-----------------
