Source: linux
Version: 5.10.223-1
Severity: important
Tags: upstream patch
X-Debbugs-Cc: debian-cl...@lists.debian.org, jaboutb...@microsoft.com
Microsoft has observed that the 5.10.y kernels in bullseye are susceptible
to crashes due to race conditions in the NVME/PCI subsystem. See below for
a representative kernel log. The problem appears most frequently in larger
systems, e.g. with 4 or more NVME devices and >= 64 CPUs, but it could
potentially occur on smaller systems as well.
The issue was fixed with the 5.14 kernel upstream in e4b9852a0 ("nvme-pci:
fix multiple races in nvme_setup_io_queues"), so this only impacts
oldstable. I have provided a backport of this commit upstream in
https://lore.kernel.org/stable/E1tj8vO-00471h-2H@lore/
I'm requesting that this commit be included in a bullseye kernel update.
Thanks
noah
SCSI subsystem initialized
[ 1.183291] pci c05b:00:00.0: 32.000 Gb/s available PCIe bandwidth, limited
by 2.5 GT/s PCIe x16 link at c05b:00:00.0 (capable of 504.112 Gb/s with 32.0
GT/s PCIe x16 link)
[ 1.191131] hv_vmbus: registering driver hv_storvsc
[ 1.191420] pci c05b:00:00.0: BAR 0: assigned [mem 0xfc0000000-0xfc007ffff
64bit]
[ 1.194446] scsi host0: storvsc_host_t
[ 1.200395] nvme nvme0: pci function c05b:00:00.0
[ 1.215047] scsi 0:0:0:2: CD-ROM Msft Virtual DVD-ROM 1.0
PQ: 0 ANSI: 5
[ 1.223119] hv_pci b8fc58bf-fbf1-4a6a-99af-6dbd17b6c699: PCI VMBus probing:
Using version 0x10003
[ 1.229528] hv_pci b8fc58bf-fbf1-4a6a-99af-6dbd17b6c699: PCI host bridge to
bus fbf1:00
[ 1.233573] pci_bus fbf1:00: root bus resource [mem 0xfc0100000-0xfc01fffff
window]
[ 1.237857] pci fbf1:00:00.0: [1414:b111] type 00 class 0x010802
[ 1.243124] pci fbf1:00:00.0: reg 0x10: [mem 0xfc0100000-0xfc01fffff 64bit]
[ 1.256727] pci fbf1:00:00.0: BAR 0: assigned [mem 0xfc0100000-0xfc01fffff
64bit]
[ 1.262177] nvme nvme1: pci function fbf1:00:00.0
[ 1.286577] hv_pci a4a2ee22-e6bb-487b-8db8-0af54d215545: PCI VMBus probing:
Using version 0x10003
[ 1.288043] BUG: unable to handle page fault for address: ffffa93540967800
[ 1.293817] hv_pci a4a2ee22-e6bb-487b-8db8-0af54d215545: PCI host bridge to
bus e6bb:00
[ 1.295878] #PF: supervisor write access in kernel mode
[ 1.295879] #PF: error_code(0x0002) - not-present page
[ 1.295880] PGD 100000067 P4D 100000067 PUD 10019e067 PMD 1011c5067 PTE 0
[ 1.295884] Oops: 0002 [#1] SMP NOPTI
[ 1.295887] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 5.10.0-25-cloud-amd64
#1 Debian 5.10.191-1
[ 1.299931] pci_bus e6bb:00: root bus resource [mem 0xfc0200000-0xfc02fffff
window]
[ 1.302446] Hardware name: Microsoft Corporation Virtual Machine/Virtual
Machine, BIOS Hyper-V UEFI Release v4.1 09/28/2024
[ 1.302451] RIP: 0010:nvme_process_cq+0x209/0x220
[ 1.302454] Code: ae f8 8b 30 89 10 0f ae f0 8b 3f 89 c8 29 f1 29 f8 83 e8
01 66 39 c8 73 ba 48 8b 03 8b 88 98 02 00 00 48 8b 43 60 48 8d 04 88 <89> 10 eb
a5 45 31 ed eb a0 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f
[ 1.306200] pci e6bb:00:00.0: [1414:b111] type 00 class 0x010802
[ 1.309039] RSP: 0018:ffffa9354022cf28 EFLAGS: 00010046
[ 1.315751] pci e6bb:00:00.0: reg 0x10: [mem 0xfc0200000-0xfc02fffff 64bit]
[ 1.318046] RAX: ffffa93540967800 RBX: ffff9461001e0000 RCX: 0000000000000200
[ 1.318047] RDX: 0000000000000002 RSI: ffffa93540017000 RDI: 0000000000000010
[ 1.318048] RBP: ffff94610d499dc0 R08: 0000000000010000 R09: 0000000000000000
[ 1.318048] R10: 0000000000000000 R11: 0000000000000000 R12: ffff947112b08010
[ 1.318049] R13: 0000000000000001 R14: 0000000000001011 R15: 0000000000000011
[ 1.318053] FS: 0000000000000000(0000) GS:ffff94807f6c0000(0000)
knlGS:0000000000000000
[ 1.344250] pci e6bb:00:00.0: BAR 0: assigned [mem 0xfc0200000-0xfc02fffff
64bit]
[ 1.352082] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.352083] CR2: ffffa93540967800 CR3: 0000000106c8c002 CR4: 0000000000370ea0
[ 1.352085] Call Trace:
[ 1.352088] <IRQ>
[ 1.352092] ? __die_body.cold+0x1a/0x1f
[ 1.358415] nvme nvme2: pci function e6bb:00:00.0
[ 1.359605] ? no_context+0x1a6/0x3c0
[ 1.380401] hv_pci 57af36b7-784f-4841-869a-cbd3bfdba787: PCI VMBus probing:
Using version 0x10003
[ 1.381687] ? exc_page_fault+0xd9/0x160
[ 1.386902] hv_pci 57af36b7-784f-4841-869a-cbd3bfdba787: PCI host bridge to
bus 784f:00
[ 1.389633] ? asm_exc_page_fault+0x1e/0x30
[ 1.392335] pci_bus 784f:00: root bus resource [mem 0xfc0300000-0xfc03fffff
window]
[ 1.395560] ? nvme_process_cq+0x209/0x220
[ 1.395563] ? nvme_process_cq+0x13d/0x220
[ 1.397733] pci 784f:00:00.0: [1414:b111] type 00 class 0x010802
[ 1.398457] nvme_irq+0x10/0x20
[ 1.398462] __handle_irq_event_percpu+0x3d/0x160
[ 1.398464] handle_irq_event+0x57/0xb0
[ 1.404076] pci 784f:00:00.0: reg 0x10: [mem 0xfc0300000-0xfc03fffff 64bit]
[ 1.404747] handle_edge_irq+0x87/0x220
[ 1.417428] pci 784f:00:00.0: BAR 0: assigned [mem 0xfc0300000-0xfc03fffff
64bit]
[ 1.418921] asm_call_irq_on_stack+0x12/0x20
[ 1.418923] </IRQ>
[ 1.418925] common_interrupt+0xb0/0x130
[ 1.418927] asm_common_interrupt+0x1e/0x40
[ 1.454390] RIP: 0010:native_safe_halt+0xe/0x20
[ 1.456577] Code: 00 f0 80 48 02 20 48 8b 00 a8 08 75 c0 e9 77 ff ff ff cc
cc cc cc cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d f6 94 55 00 fb f4 <e9> 8d 2c
35 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 e9 07 00 00
[ 1.464747] RSP: 0018:ffffa935400c3ef0 EFLAGS: 00000296
[ 1.467219] RAX: ffffffff958af2e0 RBX: 0000000000000003 RCX: ffff94807f6f4180
[ 1.467382] nvme nvme3: pci function 784f:00:00.0
[ 1.470434] RDX: 000000000000ef22 RSI: ffffa935400c3e88 RDI: 000000004add0a00
[ 1.470435] RBP: ffff946100878000 R08: 0000000000000001 R09: 000000000001a400
[ 1.470436] R10: 000000000001a400 R11: 0000000000000000 R12: 0000000000000000
[ 1.470436] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 1.470441] ? __sched_text_end+0x6/0x6
[ 1.470443] default_idle+0xa/0x20
[ 1.470445] default_idle_call+0x3c/0xd0
[ 1.491031] hv_pci 04f5eb1d-1fee-484f-b878-5de54af0fcc8: PCI VMBus probing:
Using version 0x10003
[ 1.492420] do_idle+0x20c/0x2b0
[ 1.498091] cpu_startup_entry+0x19/0x20
[ 1.499988] secondary_startup_64_no_verify+0xb0/0xbb
[ 1.502318] Modules linked in: hv_storvsc scsi_transport_fc scsi_mod
hv_netvsc pci_hyperv(+) pci_hyperv_intf hv_vmbus crc32c_intel
[ 1.507799] CR2: ffffa93540967800
[ 1.509486] ---[ end trace 62ab8a0f0ae3d1b0 ]---
[ 1.511683] RIP: 0010:nvme_process_cq+0x209/0x220
[ 1.513899] Code: ae f8 8b 30 89 10 0f ae f0 8b 3f 89 c8 29 f1 29 f8 83 e8
01 66 39 c8 73 ba 48 8b 03 8b 88 98 02 00 00 48 8b 43 60 48 8d 04 88 <89> 10 eb
a5 45 31 ed eb a0 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f
[ 1.521857] RSP: 0018:ffffa9354022cf28 EFLAGS: 00010046
[ 1.525054] RAX: ffffa93540967800 RBX: ffff9461001e0000 RCX: 0000000000000200
[ 1.528223] RDX: 0000000000000002 RSI: ffffa93540017000 RDI: 0000000000000010
[ 1.531363] RBP: ffff94610d499dc0 R08: 0000000000010000 R09: 0000000000000000
[ 1.534517] R10: 0000000000000000 R11: 0000000000000000 R12: ffff947112b08010
[ 1.537682] R13: 0000000000000001 R14: 0000000000001011 R15: 0000000000000011
[ 1.541051] FS: 0000000000000000(0000) GS:ffff94807f6c0000(0000)
knlGS:0000000000000000
[ 1.544563] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1.547224] CR2: ffffa93540967800 CR3: 0000000106c8c002 CR4: 0000000000370ea0
[ 1.550391] Kernel panic - not syncing: Fatal exception in interrupt
[ 1.562507] Kernel Offset: 0x14000000 from 0xffffffff81000000 (relocation
range: 0xffffffff80000000-0xffffffffbfffffff)
[ 1.588697] ---[ end Kernel panic - not syncing: Fatal exception in
interrupt ]---
