Control: tags -1 + moreinfo Hi Jared,
On Mon, May 08, 2023 at 11:50:21PM -0600, Jared Epp wrote: > Package: src:linux > Version: 5.10.178-3 > Severity: normal > X-Debbugs-Cc: jared...@pm.me > > Dear Maintainer, > > After I updated my Debian 11 host kernel to 5.10.0-22, my VM guest > (Windows 10 using KVM / qemu / libvirt) no longer boots and there's > a kernel null pointer dereference along with a call trace, etc. in > the system log. If I reboot and choose 5.10.0-21 in grub, the VM > works as expected and there's no error in the log. > > Below, reportbug included part of the kernel log but it missed part > of the problem so I pasted that in, I hope that's okay. If you need > any other information let me know. > > Thanks > > Jared Epp > > -- Package-specific info: > ** Version: > Linux version 5.10.0-22-amd64 (debian-kernel@lists.debian.org) (gcc-10 > (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) > #1 SMP Debian 5.10.178-3 (2023-04-22) > > ** Command line: > BOOT_IMAGE=/vmlinuz-5.10.0-22-amd64 root=/dev/mapper/panthro--vg-root ro > quiet mem_sleep_default=s2idle default_hugepagesz=1G hugepages=8 > > ** Tainted: D (128) > * kernel died recently, i.e. there was an OOPS or BUG > > ** Kernel log: > [ 51.576266] BUG: kernel NULL pointer dereference, address: 0000000000000000 > [ 51.576269] #PF: supervisor read access in kernel mode > [ 51.576270] #PF: error_code(0x0000) - not-present page > [ 51.576271] PGD 0 P4D 0 > [ 51.576273] Oops: 0000 [#1] SMP NOPTI > [ 51.576275] CPU: 6 PID: 2209 Comm: CPU 0/KVM Not tainted 5.10.0-22-amd64 > #1 Debian 5.10.178-3 > [ 51.576276] Hardware name: ASUS System Product Name/CROSSHAIR VI HERO, > BIOS 8701 02/08/2023 > [ 51.576280] RIP: 0010:find_first_bit+0x19/0x40 > [ 51.576281] Code: 5d 41 5c 41 5d 41 5e 41 5f c3 cc cc cc cc cc cc cc 49 89 > f0 48 85 f6 74 28 31 c0 eb 0d 48 83 c0 40 48 83 c7 08 4c 39 c0 73 17 <48> 8b > 17 48 85 d2 74 eb f3 48 0f bc d2 48 01 d0 49 39 c0 4c 0f 47 > [ 51.576282] RSP: 0018:ffffa99ac3a23a30 EFLAGS: 00010246 > [ 51.576283] RAX: 0000000000000000 RBX: ffffa99ac38a5000 RCX: > 0000000000000000 > [ 51.576283] RDX: 0000000000000000 RSI: 0000000000000120 RDI: > 0000000000000000 > [ 51.576284] RBP: 0000000000000000 R08: 0000000000000120 R09: > ffff94e2c1ae72a8 > [ 51.576284] R10: 000000000000000f R11: 0000000000000000 R12: > ffff94e2c1ae72a8 > [ 51.576285] R13: 0000000000000323 R14: 0000000000000003 R15: > 0000000000000006 > [ 51.576286] FS: 0000000000000000(0053) GS:ffff94e89e980000(002b) > knlGS:fffff8033f006000 > [ 51.576286] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 51.576287] CR2: 0000000000000000 CR3: 000000018e4ee000 CR4: > 0000000000750ee0 > [ 51.576287] PKRU: 55555554 > [ 51.576288] Call Trace: > [ 51.576307] kvm_make_vcpus_request_mask+0x38/0xf0 [kvm] > [ 51.576319] kvm_hv_flush_tlb+0x147/0x370 [kvm] > [ 51.576328] ? kvm_page_track_is_active+0x12/0x50 [kvm] > [ 51.576336] ? make_spte+0x146/0x260 [kvm] > [ 51.576344] ? mmu_spte_update+0x11/0x1c0 [kvm] > [ 51.576351] ? set_spte+0xee/0x140 [kvm] > [ 51.576358] ? mmu_set_spte+0x327/0x4a0 [kvm] > [ 51.576365] ? kvm_release_pfn_clean+0x22/0x40 [kvm] > [ 51.576372] ? direct_page_fault+0x223/0xa20 [kvm] > [ 51.576374] ? svm_get_segment+0x18/0x100 [kvm_amd] > [ 51.576382] ? kvm_get_cs_db_l_bits+0x35/0x70 [kvm] > [ 51.576383] ? svm_get_segment+0x18/0x100 [kvm_amd] > [ 51.576390] ? kvm_get_cs_db_l_bits+0x35/0x70 [kvm] > [ 51.576398] kvm_hv_hypercall+0x176/0x580 [kvm] > [ 51.576401] ? get_cpu_vendor+0x40/0xa0 > [ 51.576403] ? native_load_tr_desc+0x67/0x70 > [ 51.576411] kvm_arch_vcpu_ioctl_run+0xbe8/0x1740 [kvm] > [ 51.576419] kvm_vcpu_ioctl+0x21e/0x5b0 [kvm] > [ 51.576422] __x64_sys_ioctl+0x8b/0xc0 > [ 51.576424] do_syscall_64+0x33/0x80 > [ 51.576426] entry_SYSCALL_64_after_hwframe+0x61/0xc6 > [ 51.576428] RIP: 0033:0x7fad816f2237 > [ 51.576429] Code: 00 00 00 48 8b 05 59 cc 0d 00 64 c7 00 26 00 00 00 48 c7 > c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d > 01 f0 ff ff 73 01 c3 48 8b 0d 29 cc 0d 00 f7 d8 64 89 01 48 > [ 51.576429] RSP: 002b:00007fad7ce65508 EFLAGS: 00000246 ORIG_RAX: > 0000000000000010 > [ 51.576430] RAX: ffffffffffffffda RBX: 000000000000ae80 RCX: > 00007fad816f2237 > [ 51.576431] RDX: 0000000000000000 RSI: 000000000000ae80 RDI: > 000000000000001c > [ 51.576431] RBP: 000055a3e17511c0 R08: 000055a3df109848 R09: > 000055a3df5335c0 > [ 51.576432] R10: 0000000000000000 R11: 0000000000000246 R12: > 0000000000000000 > [ 51.576432] R13: 000055a3df54fbc0 R14: 00007fad7ce657c0 R15: > 0000000000802000 > [ 51.576434] Modules linked in: xt_nat veth nft_chain_nat xt_MASQUERADE > nf_nat nf_conntrack_netlink xfrm_user xfrm_algo br_netfilter vhost_net vhost > vhost_iotlb tap tun bridge stp llc overlay ip6t_REJECT nf_reject_ipv6 xt_hl > ip6_tables ip6t_rt ipt_REJECT nf_reject_ipv4 xt_multiport nft_limit > snd_hda_codec_realtek snd_hda_codec_generic ledtrig_audio snd_hda_codec_hdmi > snd_hda_intel nls_ascii snd_intel_dspcfg nls_cp437 soundwire_intel vfat > soundwire_generic_allocation fat snd_soc_core snd_compress soundwire_cadence > snd_hda_codec edac_mce_amd xt_limit xt_addrtype kvm_amd snd_hda_core > xt_tcpudp snd_hwdep eeepc_wmi kvm soundwire_bus xpad xt_conntrack cdc_acm > joydev asus_wmi ff_memless snd_pcm nf_conntrack battery sparse_keymap > snd_timer nf_defrag_ipv6 rfkill nf_defrag_ipv4 irqbypass nft_compat snd video > rapl efi_pstore wmi_bmof pcspkr ccp soundcore k10temp sp5100_tco nft_counter > watchdog sg tpm_crb tpm_tis tpm_tis_core tpm rng_core acpi_cpufreq evdev > nf_tables libcrc32c nfnetlink msr fuse > [ 51.576464] configfs efivarfs ip_tables x_tables autofs4 ext4 crc16 > mbcache jbd2 crc32c_generic dm_crypt dm_mod hid_logitech_hidpp > hid_logitech_dj amdgpu sr_mod cdrom gpu_sched sd_mod hid_generic ttm > crc32_pclmul crc32c_intel usbhid hid drm_kms_helper ahci cec libahci > ghash_clmulni_intel xhci_pci libata xhci_hcd drm nvme aesni_intel mxm_wmi igb > libaes usbcore crypto_simd nvme_core cryptd scsi_mod glue_helper i2c_piix4 > dca ptp pps_core t10_pi i2c_algo_bit crc_t10dif crct10dif_generic usb_common > crct10dif_pclmul crct10dif_common wmi gpio_amdpt gpio_generic button > [ 51.576484] CR2: 0000000000000000 > [ 51.576485] ---[ end trace acfac62cc884c67c ]--- > [ 51.668091] pstore: crypto_comp_compress failed, ret = -22! > [ 51.682455] br-b8df22c12cd5: port 4(vethe67c4df) entered blocking state > [ 51.682459] br-b8df22c12cd5: port 4(vethe67c4df) entered disabled state > [ 51.682501] device vethe67c4df entered promiscuous mode > [ 51.689861] br-b8df22c12cd5: port 4(vethe67c4df) entered blocking state > [ 51.689863] br-b8df22c12cd5: port 4(vethe67c4df) entered forwarding state > [ 51.696372] RIP: 0010:find_first_bit+0x19/0x40 > [ 51.696374] Code: 5d 41 5c 41 5d 41 5e 41 5f c3 cc cc cc cc cc cc cc 49 89 > f0 48 85 f6 74 28 31 c0 eb 0d 48 83 c0 40 48 83 c7 08 4c 39 c0 73 17 <48> 8b > 17 48 85 d2 74 eb f3 48 0f bc d2 48 01 d0 49 39 c0 4c 0f 47 > [ 51.696376] RSP: 0018:ffffa99ac3a23a30 EFLAGS: 00010246 > [ 51.696378] RAX: 0000000000000000 RBX: ffffa99ac38a5000 RCX: > 0000000000000000 > [ 51.696379] RDX: 0000000000000000 RSI: 0000000000000120 RDI: > 0000000000000000 > [ 51.696380] RBP: 0000000000000000 R08: 0000000000000120 R09: > ffff94e2c1ae72a8 > [ 51.696380] R10: 000000000000000f R11: 0000000000000000 R12: > ffff94e2c1ae72a8 > [ 51.696381] R13: 0000000000000323 R14: 0000000000000003 R15: > 0000000000000006 > [ 51.696383] FS: 0000000000000000(0053) GS:ffff94e89e980000(002b) > knlGS:fffff8033f006000 > [ 51.696384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 51.696384] CR2: 0000000000000000 CR3: 000000018e4ee000 CR4: > 0000000000750ee0 > [ 51.696385] PKRU: 55555554 > [ 51.700146] br-924f74569f8a: port 4(veth5e7362a) entered blocking state > [ 51.700151] br-924f74569f8a: port 4(veth5e7362a) entered disabled state > [ 51.700200] device veth5e7362a entered promiscuous mode > [ 51.700257] br-924f74569f8a: port 4(veth5e7362a) entered blocking state > [ 51.700259] br-924f74569f8a: port 4(veth5e7362a) entered forwarding state > [ 51.787480] eth0: renamed from veth28831ed > [ 51.831676] br-b8df22c12cd5: port 4(vethe67c4df) entered disabled state > [ 51.831721] br-924f74569f8a: port 4(veth5e7362a) entered disabled state > [ 51.831741] IPv6: ADDRCONF(NETDEV_CHANGE): veth25ca080: link becomes ready > [ 51.831758] br-924f74569f8a: port 2(veth25ca080) entered blocking state > [ 51.831759] br-924f74569f8a: port 2(veth25ca080) entered forwarding state > [ 51.832280] br-b8df22c12cd5: port 5(vethc8c90d8) entered blocking state > [ 51.832282] br-b8df22c12cd5: port 5(vethc8c90d8) entered disabled state > [ 51.832329] device vethc8c90d8 entered promiscuous mode > [ 51.832383] br-b8df22c12cd5: port 5(vethc8c90d8) entered blocking state > [ 51.832385] br-b8df22c12cd5: port 5(vethc8c90d8) entered forwarding state > [ 51.832416] br-924f74569f8a: port 5(vethfbf8266) entered blocking state > [ 51.832418] br-924f74569f8a: port 5(vethfbf8266) entered disabled state > [ 51.832452] device vethfbf8266 entered promiscuous mode > [ 51.832503] br-924f74569f8a: port 5(vethfbf8266) entered blocking state > [ 51.832504] br-924f74569f8a: port 5(vethfbf8266) entered forwarding state > [ 51.955355] eth0: renamed from vethec69e8f > [ 51.999437] eth0: renamed from veth30923c8 > [ 52.043965] br-b8df22c12cd5: port 5(vethc8c90d8) entered disabled state > [ 52.044034] br-924f74569f8a: port 5(vethfbf8266) entered disabled state > [ 52.044064] IPv6: ADDRCONF(NETDEV_CHANGE): veth1661ccb: link becomes ready > [ 52.044086] br-924f74569f8a: port 1(veth1661ccb) entered blocking state > [ 52.044088] br-924f74569f8a: port 1(veth1661ccb) entered forwarding state > [ 52.044108] IPv6: ADDRCONF(NETDEV_CHANGE): veth0ebab0a: link becomes ready > [ 52.044125] br-b8df22c12cd5: port 1(veth0ebab0a) entered blocking state > [ 52.044127] br-b8df22c12cd5: port 1(veth0ebab0a) entered forwarding state > [ 52.044539] br-924f74569f8a: port 6(veth5372881) entered blocking state > [ 52.044542] br-924f74569f8a: port 6(veth5372881) entered disabled state > [ 52.044586] device veth5372881 entered promiscuous mode > [ 52.044644] br-924f74569f8a: port 6(veth5372881) entered blocking state > [ 52.044646] br-924f74569f8a: port 6(veth5372881) entered forwarding state > [ 52.057025] br-924f74569f8a: port 7(veth29a4d93) entered blocking state > [ 52.057028] br-924f74569f8a: port 7(veth29a4d93) entered disabled state > [ 52.057108] device veth29a4d93 entered promiscuous mode > [ 52.057175] br-924f74569f8a: port 7(veth29a4d93) entered blocking state > [ 52.057176] br-924f74569f8a: port 7(veth29a4d93) entered forwarding state > [ 52.231474] eth0: renamed from veth9d75af1 > [ 52.255847] br-924f74569f8a: port 6(veth5372881) entered disabled state > [ 52.255889] br-924f74569f8a: port 7(veth29a4d93) entered disabled state > [ 52.255906] IPv6: ADDRCONF(NETDEV_CHANGE): veth5e7362a: link becomes ready > [ 52.255928] br-924f74569f8a: port 4(veth5e7362a) entered blocking state > [ 52.255929] br-924f74569f8a: port 4(veth5e7362a) entered forwarding state > [ 52.347639] eth0: renamed from veth0803979 > [ 52.419482] eth0: renamed from veth7d2c7ef > [ 52.508188] eth1: renamed from vethee5bda2 > [ 52.567361] IPv6: ADDRCONF(NETDEV_CHANGE): veth74b84d1: link becomes ready > [ 52.567383] br-b8df22c12cd5: port 3(veth74b84d1) entered blocking state > [ 52.567385] br-b8df22c12cd5: port 3(veth74b84d1) entered forwarding state > [ 52.567531] IPv6: ADDRCONF(NETDEV_CHANGE): veth226976d: link becomes ready > [ 52.567550] br-b8df22c12cd5: port 2(veth226976d) entered blocking state > [ 52.567551] br-b8df22c12cd5: port 2(veth226976d) entered forwarding state > [ 52.567730] IPv6: ADDRCONF(NETDEV_CHANGE): veth5372881: link becomes ready > [ 52.567751] br-924f74569f8a: port 6(veth5372881) entered blocking state > [ 52.567752] br-924f74569f8a: port 6(veth5372881) entered forwarding state > [ 52.615463] eth1: renamed from vethef7b9c6 > [ 52.643620] IPv6: ADDRCONF(NETDEV_CHANGE): vethfbf8266: link becomes ready > [ 52.643649] br-924f74569f8a: port 5(vethfbf8266) entered blocking state > [ 52.643652] br-924f74569f8a: port 5(vethfbf8266) entered forwarding state > [ 52.690903] eth0: renamed from veth3feaf4a > [ 52.723412] IPv6: ADDRCONF(NETDEV_CHANGE): vethc8c90d8: link becomes ready > [ 52.723440] br-b8df22c12cd5: port 5(vethc8c90d8) entered blocking state > [ 52.723442] br-b8df22c12cd5: port 5(vethc8c90d8) entered forwarding state > [ 52.847525] eth1: renamed from vethd1e1e47 > [ 52.877625] IPv6: ADDRCONF(NETDEV_CHANGE): veth888282e: link becomes ready > [ 52.877656] br-924f74569f8a: port 3(veth888282e) entered blocking state > [ 52.877660] br-924f74569f8a: port 3(veth888282e) entered forwarding state > [ 52.877676] eth0: renamed from veth374d91f > [ 52.939600] IPv6: ADDRCONF(NETDEV_CHANGE): vethe67c4df: link becomes ready > [ 52.939638] br-b8df22c12cd5: port 4(vethe67c4df) entered blocking state > [ 52.939641] br-b8df22c12cd5: port 4(vethe67c4df) entered forwarding state > [ 52.945134] eth1: renamed from veth583ed90 > [ 52.971727] IPv6: ADDRCONF(NETDEV_CHANGE): veth29a4d93: link becomes ready > [ 52.971776] br-924f74569f8a: port 7(veth29a4d93) entered blocking state > [ 52.971778] br-924f74569f8a: port 7(veth29a4d93) entered forwarding state > [ 60.145487] logitech-hidpp-device 0003:046D:4031.0007: HID++ 2.0 device > connected. > [ 63.847187] bridge0: port 1(vnet0) entered learning state > [ 65.643192] bridge0: port 2(enp5s0) entered learning state > [ 67.654690] kauditd_printk_skb: 50 callbacks suppressed > [ 67.654691] audit: type=1400 audit(1683610301.229:62): apparmor="DENIED" > operation="capable" profile="libvirtd" pid=1754 comm="prio-rpc-worker" > capability=17 capname="sys_rawio" > [ 78.951195] bridge0: port 1(vnet0) entered forwarding state > [ 78.951211] bridge0: topology change detected, propagating > [ 80.743422] bridge0: port 2(enp5s0) entered forwarding state > [ 80.743435] bridge0: topology change detected, propagating This sounds similar to the https://forum.proxmox.com/threads/with-latest-5-15-104-1-pve-windows-server-vm-freeze-stuck.125294/ issue. Would you be able to verify two things: Check how the Windows VM is configured and if you pass the '+hv-tlbflush' flag. Additionally, would the attached patch make the issue go away? Regards, Salvatore