** Description changed:
- == Comment: #0 - SEETEENA THOUFEEK <sthou...@in.ibm.com> - 2025-03-28
04:21:14 ==
- +++ This bug was initially created as a clone of Bug #209721 +++
+ [Impact]
- ---Problem Description---
- WARNING: CPU: 18 PID: 3683 at arch/powerpc/kvm/../../../virt/kvm/vfio.c Call
Traces seen when pci device is detached from the kvm guest
-
+ Currently on book3s-hv, the capability KVM_CAP_SPAPR_TCE_VFIO is only
+ available for KVM Guests running on PowerNV and not for the KVM guests
+ running on pSeries hypervisors. This prevents a pSeries L2 guest from
+ leveraging the in-kernel acceleration for H_PUT_TCE_INDIRECT and
+ H_STUFF_TCE hcalls that results in slow startup times for large memory
+ guests.
+
+ Support for VFIO on pSeries was restored in commit f431a8cde7f1
+ ("powerpc/iommu: Reimplement the iommu_table_group_ops for pSeries"),
+ making it possible to re-enable this capability on pSeries hosts.
+
+ This change enables KVM_CAP_SPAPR_TCE_VFIO for nested PAPR guests on
+ pSeries, while maintaining the existing behavior on PowerNV. Booting an
+ L2 guest with 128GB of memory shows an average 11% improvement in
+ startup time.
+ [Fix]
- --- Steps to reproduce ---
- 1. Start a guest
+ Clean cherry pick of:
+ - b4392813bbc3b0 KVM: PPC: Enable CAP_SPAPR_TCE_VFIO on pSeries KVM guests
+ from linux-next.
+
+ [Test Plan]
+
+ Tested IBM side using the following procedure:
+ 1. Start a guest
2. unbind the pci device from host using below command
root@ltcrain8og-lp6:~/NET# virsh nodedev-detach pci_0028_01_00_0
Device pci_0028_01_00_0 detached
root@ltcrain8og-lp6:~/NET# virsh nodedev-detach pci_0028_01_00_1
Device pci_0028_01_00_1 detached
3. Now attach the pci device to guest using virsh attach-device command.
The xml snippet used is given below
root@ltcrain8og-lp6:~/NET# vi pci2.xml
<hostdev mode='subsystem' type='pci' managed='yes'>
- <source>
- <address domain='0x0028' bus='0x01' slot='0x00' function='0x0'/>
- </source>
+ <source>
+ <address domain='0x0028' bus='0x01' slot='0x00' function='0x0'/>
+ </source>
</hostdev>
- root@ltcrain8og-lp6:~/NET# virsh attach-device vm pci2.xml
+ root@ltcrain8og-lp6:~/NET# virsh attach-device vm pci2.xml
Device attached successfully
4. On guest side the pci device gets attached sucesfully, and interface comes
up.
[root@localhost ~]# lspci
0000:00:01.0 Ethernet controller: Red Hat, Inc. Virtio network device
0000:00:02.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
0000:00:03.0 USB controller: Red Hat, Inc. QEMU XHCI Host Controller (rev 01)
0000:00:04.0 Communication controller: Red Hat, Inc. Virtio console
0000:00:05.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon
0000:00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG
0001:00:01.0 Ethernet controller: Mellanox Technologies MT27710 Family
[ConnectX-4 Lx]
5. Now Detach the device using virsh detach command
- root@ltcrain8og-lp6:~/NET# virsh detach-device vm pci2.xml
+ root@ltcrain8og-lp6:~/NET# virsh detach-device vm pci2.xml
+ Device detached successfully
+
+ [Where problems could occur]
+
+ The fix affects the KVM subsystem on IBM Power platforms.
+ An issue with this fix may lead to incorrect exposure of the
+ KVM_CAP_SPAPR_TCE_VFIO capability on hosts or guests
+ that do not fully support it.
+
+ -----
+
+ == Comment: #0 - SEETEENA THOUFEEK <sthou...@in.ibm.com> - 2025-03-28
04:21:14 ==
+ +++ This bug was initially created as a clone of Bug #209721 +++
+
+ ---Problem Description---
+ WARNING: CPU: 18 PID: 3683 at arch/powerpc/kvm/../../../virt/kvm/vfio.c Call
Traces seen when pci device is detached from the kvm guest
+
+ --- Steps to reproduce ---
+ 1. Start a guest
+ 2. unbind the pci device from host using below command
+
+ root@ltcrain8og-lp6:~/NET# virsh nodedev-detach pci_0028_01_00_0
+ Device pci_0028_01_00_0 detached
+ root@ltcrain8og-lp6:~/NET# virsh nodedev-detach pci_0028_01_00_1
+ Device pci_0028_01_00_1 detached
+
+ 3. Now attach the pci device to guest using virsh attach-device command.
+ The xml snippet used is given below
+
+ root@ltcrain8og-lp6:~/NET# vi pci2.xml
+ <hostdev mode='subsystem' type='pci' managed='yes'>
+ <source>
+ <address domain='0x0028' bus='0x01' slot='0x00' function='0x0'/>
+ </source>
+ </hostdev>
+
+ root@ltcrain8og-lp6:~/NET# virsh attach-device vm pci2.xml
+ Device attached successfully
+
+ 4. On guest side the pci device gets attached sucesfully, and interface comes
up.
+ [root@localhost ~]# lspci
+ 0000:00:01.0 Ethernet controller: Red Hat, Inc. Virtio network device
+ 0000:00:02.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
+ 0000:00:03.0 USB controller: Red Hat, Inc. QEMU XHCI Host Controller (rev 01)
+ 0000:00:04.0 Communication controller: Red Hat, Inc. Virtio console
+ 0000:00:05.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon
+ 0000:00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio RNG
+ 0001:00:01.0 Ethernet controller: Mellanox Technologies MT27710 Family
[ConnectX-4 Lx]
+
+ 5. Now Detach the device using virsh detach command
+ root@ltcrain8og-lp6:~/NET# virsh detach-device vm pci2.xml
Device detached successfully
[root@ltcrain8og-lp6 ~]# [ 493.501609] ------------[ cut here ]------------
[ 493.501632] WARNING: CPU: 18 PID: 3683 at
arch/powerpc/kvm/../../../virt/kvm/vfio.c:103 kvm_vfio_set_attr+0x810/0x8c0
[kvm]
[ 493.501653] Modules linked in: vfio_pci vfio_pci_core vfio_iommu_spapr_tce
vfio iommufd vhost_net vhost vhost_iotlb tap nft_masq nft_ct nft_reject_ipv4
nf_reject_ipv4 nft_reject nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6
nf_defrag_ipv4 bridge stp llc tun kvm_hv kvm rpadlpar_io rpaphp xsk_diag
vsock_diag bonding rfkill nf_tables sunrpc mlx5_ib ib_uverbs macsec ib_core
mlx5_core mlxfw psample tls tg3 vmx_crypto ibmveth aes_gcm_p10_crypto
crct10dif_vpmsum binfmt_misc pseries_rng fuse loop nfnetlink vsock_loopback
vmw_vsock_virtio_transport_common vsock zram xfs dm_service_time nvme ibmvfc
ibmvscsi nvme_core scsi_transport_fc scsi_transport_srp crc32c_vpmsum
pseries_wdt nvme_auth scsi_dh_rdac scsi_dh_emc scsi_dh_alua dm_multipath
[ 493.501712] CPU: 18 UID: 0 PID: 3683 Comm: qemu-system-ppc Kdump: loaded
Not tainted 6.11.0-63.fc41.ppc64le #1
[ 493.501717] Hardware name: IBM,9105-22A POWER10 (architected) 0x800200
0xf000006 of:IBM,FW1060.11 (NL1060_066) hv:phyp pSeries
[ 493.501724] NIP: c0080000076c5bc8 LR: c0080000076c597c CTR:
c000000000bcb820
[ 493.501730] REGS: c00000005ec53730 TRAP: 0700 Not tainted
(6.11.0-63.fc41.ppc64le)
[ 493.501736] MSR: 8000000000029033 <SF,EE,ME,IR,DR,RI,LE> CR: 28004428
XER: 00000000
- [ 493.501746] CFAR: c0080000076c59c4 IRQMASK: 0
- [ 493.501746] GPR00: c0080000076c575c c00000005ec539d0 c008000005991200
c000000111460000
- [ 493.501746] GPR04: 0000000000000000 c00000004a9e0d60 c00000004aa16480
c000000045618000
- [ 493.501746] GPR08: 000000005deadbee 5deadbeef0000100 5deadbeef0000122
c0080000076e3778
- [ 493.501746] GPR12: c000000000bcb820 c000000c7de7b700 0000000000000000
0000000000000000
- [ 493.501746] GPR16: 0000000000000000 0000000000000000 0000000000000000
0000000000000000
- [ 493.501746] GPR20: 0000000000000000 0000000000000000 c000000014a68c00
000000000000008c
- [ 493.501746] GPR24: 000000008018aee1 000000000000008c c00000004aa16490
c0000000ccbaf800
- [ 493.501746] GPR28: c0000000ccbaf801 c00000004aa16480 c00000004aa1d140
c00000004a9e0d60
+ [ 493.501746] CFAR: c0080000076c59c4 IRQMASK: 0
+ [ 493.501746] GPR00: c0080000076c575c c00000005ec539d0 c008000005991200
c000000111460000
+ [ 493.501746] GPR04: 0000000000000000 c00000004a9e0d60 c00000004aa16480
c000000045618000
+ [ 493.501746] GPR08: 000000005deadbee 5deadbeef0000100 5deadbeef0000122
c0080000076e3778
+ [ 493.501746] GPR12: c000000000bcb820 c000000c7de7b700 0000000000000000
0000000000000000
+ [ 493.501746] GPR16: 0000000000000000 0000000000000000 0000000000000000
0000000000000000
+ [ 493.501746] GPR20: 0000000000000000 0000000000000000 c000000014a68c00
000000000000008c
+ [ 493.501746] GPR24: 000000008018aee1 000000000000008c c00000004aa16490
c0000000ccbaf800
+ [ 493.501746] GPR28: c0000000ccbaf801 c00000004aa16480 c00000004aa1d140
c00000004a9e0d60
[ 493.501803] NIP [c0080000076c5bc8] kvm_vfio_set_attr+0x810/0x8c0 [kvm]
[ 493.501819] LR [c0080000076c597c] kvm_vfio_set_attr+0x5c4/0x8c0 [kvm]
[ 493.501834] Call Trace:
[ 493.501836] [c00000005ec539d0] [c0080000076c575c]
kvm_vfio_set_attr+0x3a4/0x8c0 [kvm] (unreliable)
[ 493.501852] [c00000005ec53a90] [c0080000076b7960]
kvm_device_ioctl+0xd8/0x1c0 [kvm]
[ 493.501870] [c00000005ec53ae0] [c00000000069c154] sys_ioctl+0x574/0x16a0
[ 493.501878] [c00000005ec53be0] [c000000000030830]
system_call_exception+0x160/0x310
[ 493.501888] [c00000005ec53e50] [c00000000000d05c]
system_call_vectored_common+0x15c/0x2ec
[ 493.501900] --- interrupt: 3000 at 0x7fffb8b2d53c
[ 493.501907] NIP: 00007fffb8b2d53c LR: 00007fffb8b2d53c CTR:
0000000000000000
[ 493.501912] REGS: c00000005ec53e80 TRAP: 3000 Not tainted
(6.11.0-63.fc41.ppc64le)
[ 493.501916] MSR: 800000000280f033 <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE>
CR: 42004422 XER: 00000000
- [ 493.501927] IRQMASK: 0
- [ 493.501927] GPR00: 0000000000000036 00007fffb720d900 00007fffb720d990
000000000000008c
- [ 493.501927] GPR04: 000000008018aee1 00007fffb720d990 0000000000000000
0000000000000002
- [ 493.501927] GPR08: 000000000000008c 0000000000000000 0000000000000000
0000000000000000
- [ 493.501927] GPR12: 0000000000000000 00007fffb7215ea0 0000000000000000
00007fffb720e7a0
- [ 493.501927] GPR16: 0000000000000000 00007fffb720eea0 0000000000000000
0000000000000000
- [ 493.501927] GPR20: 00007fffbaba0000 0000000000001160 000000000080dfa0
00007fffb6a00000
- [ 493.501927] GPR24: 0000000000000000 0000000177586ac0 00007fffb720db00
000000013ed6eea0
- [ 493.501927] GPR28: 0000000000000000 0000000177a21100 0000000000000006
00007fffb720da20
+ [ 493.501927] IRQMASK: 0
+ [ 493.501927] GPR00: 0000000000000036 00007fffb720d900 00007fffb720d990
000000000000008c
+ [ 493.501927] GPR04: 000000008018aee1 00007fffb720d990 0000000000000000
0000000000000002
+ [ 493.501927] GPR08: 000000000000008c 0000000000000000 0000000000000000
0000000000000000
+ [ 493.501927] GPR12: 0000000000000000 00007fffb7215ea0 0000000000000000
00007fffb720e7a0
+ [ 493.501927] GPR16: 0000000000000000 00007fffb720eea0 0000000000000000
0000000000000000
+ [ 493.501927] GPR20: 00007fffbaba0000 0000000000001160 000000000080dfa0
00007fffb6a00000
+ [ 493.501927] GPR24: 0000000000000000 0000000177586ac0 00007fffb720db00
000000013ed6eea0
+ [ 493.501927] GPR28: 0000000000000000 0000000177a21100 0000000000000006
00007fffb720da20
[ 493.501967] NIP [00007fffb8b2d53c] 0x7fffb8b2d53c
[ 493.501972] LR [00007fffb8b2d53c] 0x7fffb8b2d53c
[ 493.501977] --- interrupt: 3000
- [ 493.501980] Code: 4bfffd34 60420000 60000000 e80100d0 eba100a8 7c0803a6
4bfff874 3be00000 3ba0fff2 4bfffe94 60000000 4bffffe0 <0fe00000> 4bfffe18
f99f0018 4bfffadc
+ [ 493.501980] Code: 4bfffd34 60420000 60000000 e80100d0 eba100a8 7c0803a6
4bfff874 3be00000 3ba0fff2 4bfffe94 60000000 4bffffe0 <0fe00000> 4bfffe18
f99f0018 4bfffadc
[ 493.502000] ---[ end trace 0000000000000000 ]---
-
- NOTE : The call traces are seen only for the first time the device is
detached. For second time on wards there are no call traces seen. Also when the
lpar is freshly started after a reboot and then the above steps are performed.
+ NOTE : The call traces are seen only for the first time the device is
+ detached. For second time on wards there are no call traces seen. Also
+ when the lpar is freshly started after a reboot and then the above steps
+ are performed.
---Debugger---
A debugger is not configured
-
- Machine Type = na
-
+
+ Machine Type = na
+
---uname output---
na
-
+
Contact Information = sthou...@in.ibm.com
-
The patch is now merged in upstream Linux:
- b4392813bbc3 ("KVM: PPC: Enable CAP_SPAPR_TCE_VFIO on pSeries KVM
+ b4392813bbc3 ("KVM: PPC: Enable CAP_SPAPR_TCE_VFIO on pSeries KVM
guests")
~Amit
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2104893
Title:
WARNING: CPU: 18 PID: 3683 at
arch/powerpc/kvm/../../../virt/kvm/vfio.c Call Traces seen when pci
device is detached from the kvm guest
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/2104893/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
