On 6/4/2025 7:34 PM, Donald Dutile wrote:
On 6/4/25 10:02 PM, Nathan Chen wrote:
On 6/2/2025 8:41 AM, Shameer Kolothum wrote:
This patch series introduces support for a user-creatable SMMUv3 device
(-device arm-smmuv3) in QEMU.
Tested-by: Nathan Chen <nath...@nvidia.com>
I am able to create 16 SMMUv3 devices in a qemu VM with emulated
devices properly associated with the guest SMMUs in guest sysfs -
verified with some guest SMMUs having two or three emulated NICs
assigned to them while other guest SMMUs have a minimum of one assigned.
Thanks,
Nathan
Nathan,
Great test!
Can you share the xml &/or qemu cmdline, to demonstrate this broad
capability?
Also would be a good reference when (we know it isn't 'if') a new/
different config
that wasn't tested shows a bug, and we'll have more info to work with.
Yes, here is the qemu command line I used. smmuv3.15 is associated with
two devices, smmuv3.0 is associated with two devices, and smmuv3.1 is
associated with three devices:
qemu-system-aarch64 \
-machine hmat=on -machine virt,accel=kvm,gic-version=3,ras=on \
-cpu host -smp cpus=4 -m size=16G,slots=4,maxmem=32G -nographic \
-bios /usr/share/AAVMF/AAVMF_CODE.fd \
-device nvme,drive=nvme0,serial=deadbeaf1,bus=pcie.0 \
-drive
file=/home/nvidia/nathanc/noble-server-cloudimg-arm64.qcow2,index=0,media=disk,format=qcow2,if=none,id=nvme0
\
-device
e1000,romfile=/root/nathanc/efi-e1000.rom,netdev=net0,bus=pcie.0 \
-netdev
user,id=net0,hostfwd=tcp::5558-:22,hostfwd=tcp::5586-:5586 \
-netdev user,id=net1 \
-netdev user,id=net2 \
-netdev user,id=net3 \
-netdev user,id=net4 \
-netdev user,id=net5 \
-netdev user,id=net6 \
-netdev user,id=net7 \
-netdev user,id=net8 \
-netdev user,id=net9 \
-netdev user,id=net10 \
-netdev user,id=net11 \
-netdev user,id=net12 \
-netdev user,id=net13 \
-netdev user,id=net14 \
-netdev user,id=net15 \
-netdev user,id=net16 \
-netdev user,id=net17 \
-netdev user,id=net18 \
-netdev user,id=net19 \
-netdev user,id=net20 \
-device arm-smmuv3,primary-bus=pcie.0,id=smmuv3.0 \
-device virtio-net-pci,bus=pcie.0,netdev=net1 \
-device virtio-net-pci,bus=pcie.0,netdev=net20 \
-device pxb-pcie,id=pcie.1,bus_nr=200,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.1,id=smmuv3.1 \
-device pcie-root-port,id=pcie.port1,bus=pcie.1,slot=1,chassis=1 \
-device virtio-net-pci,bus=pcie.port1,netdev=net2 \
-device
pcie-root-port,id=pcie.port17,bus=pcie.1,slot=17,chassis=17 \
-device virtio-net-pci,bus=pcie.port17,netdev=net18 \
-device
pcie-root-port,id=pcie.port18,bus=pcie.1,slot=18,chassis=18 \
-device virtio-net-pci,bus=pcie.port18,netdev=net19 \
-device pxb-pcie,id=pcie.2,bus_nr=196,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.2,id=smmuv3.2 \
-device pcie-root-port,id=pcie.port2,bus=pcie.2,slot=2,chassis=2 \
-device virtio-net-pci,bus=pcie.port2,netdev=net3 \
-device pxb-pcie,id=pcie.3,bus_nr=192,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.3,id=smmuv3.3 \
-device pcie-root-port,id=pcie.port3,bus=pcie.3,slot=3,chassis=3 \
-device virtio-net-pci,bus=pcie.port3,netdev=net4 \
-device pxb-pcie,id=pcie.4,bus_nr=188,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.4,id=smmuv3.4 \
-device pcie-root-port,id=pcie.port4,bus=pcie.4,slot=4,chassis=4 \
-device virtio-net-pci,bus=pcie.port4,netdev=net5 \
-device pxb-pcie,id=pcie.5,bus_nr=184,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.5,id=smmuv3.5 \
-device pcie-root-port,id=pcie.port5,bus=pcie.5,slot=5,chassis=5 \
-device virtio-net-pci,bus=pcie.port5,netdev=net6 \
-device pxb-pcie,id=pcie.6,bus_nr=180,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.6,id=smmuv3.6 \
-device pcie-root-port,id=pcie.port6,bus=pcie.6,slot=6,chassis=6 \
-device virtio-net-pci,bus=pcie.port6,netdev=net7 \
-device pxb-pcie,id=pcie.7,bus_nr=176,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.7,id=smmuv3.7 \
-device pcie-root-port,id=pcie.port7,bus=pcie.7,slot=7,chassis=7 \
-device virtio-net-pci,bus=pcie.port7,netdev=net8 \
-device pxb-pcie,id=pcie.8,bus_nr=172,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.8,id=smmuv3.8 \
-device pcie-root-port,id=pcie.port8,bus=pcie.8,slot=8,chassis=8 \
-device virtio-net-pci,bus=pcie.port8,netdev=net9 \
-device pxb-pcie,id=pcie.9,bus_nr=168,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.9,id=smmuv3.9 \
-device pcie-root-port,id=pcie.port9,bus=pcie.9,slot=9,chassis=9 \
-device virtio-net-pci,bus=pcie.port9,netdev=net10 \
-device pxb-pcie,id=pcie.10,bus_nr=164,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.10,id=smmuv3.10 \
-device
pcie-root-port,id=pcie.port10,bus=pcie.10,slot=10,chassis=10 \
-device virtio-net-pci,bus=pcie.port10,netdev=net11 \
-device pxb-pcie,id=pcie.11,bus_nr=160,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.11,id=smmuv3.11 \
-device
pcie-root-port,id=pcie.port11,bus=pcie.11,slot=11,chassis=11 \
-device virtio-net-pci,bus=pcie.port11,netdev=net12 \
-device pxb-pcie,id=pcie.12,bus_nr=156,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.12,id=smmuv3.12 \
-device
pcie-root-port,id=pcie.port12,bus=pcie.12,slot=12,chassis=12 \
-device virtio-net-pci,bus=pcie.port12,netdev=net13 \
-device pxb-pcie,id=pcie.13,bus_nr=152,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.13,id=smmuv3.13 \
-device
pcie-root-port,id=pcie.port13,bus=pcie.13,slot=13,chassis=13 \
-device virtio-net-pci,bus=pcie.port13,netdev=net14 \
-device pxb-pcie,id=pcie.14,bus_nr=148,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.14,id=smmuv3.14 \
-device
pcie-root-port,id=pcie.port14,bus=pcie.14,slot=14,chassis=14 \
-device virtio-net-pci,bus=pcie.port14,netdev=net15 \
-device pxb-pcie,id=pcie.15,bus_nr=144,bus=pcie.0 \
-device arm-smmuv3,primary-bus=pcie.15,id=smmuv3.15 \
-device
pcie-root-port,id=pcie.port15,bus=pcie.15,slot=15,chassis=15 \
-device virtio-net-pci,bus=pcie.port15,netdev=net16 \
-device
pcie-root-port,id=pcie.port16,bus=pcie.15,slot=16,chassis=16 \
-device virtio-net-pci,bus=pcie.port16,netdev=net17
Here is the guest topology:
$ lspci -tv
-+-[0000:00]-+-00.0 Red Hat, Inc. QEMU PCIe Host bridge
| +-01.0 Red Hat, Inc. QEMU NVM Express Controller
| +-02.0 Intel Corporation 82540EM Gigabit Ethernet Controller
| +-03.0 Red Hat, Inc. Virtio network device
| +-04.0 Red Hat, Inc. Virtio network device
| +-05.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-06.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-07.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-08.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-09.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-0a.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-0b.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-0c.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-0d.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-0e.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-0f.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-10.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-11.0 Red Hat, Inc. QEMU PCIe Expander bridge
| +-12.0 Red Hat, Inc. QEMU PCIe Expander bridge
| \-13.0 Red Hat, Inc. QEMU PCIe Expander bridge
+-[0000:90]-+-00.0-[91]----00.0 Red Hat, Inc. Virtio 1.0 network device
| \-01.0-[92]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:94]---00.0-[95]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:98]---00.0-[99]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:9c]---00.0-[9d]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:a0]---00.0-[a1]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:a4]---00.0-[a5]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:a8]---00.0-[a9]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:ac]---00.0-[ad]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:b0]---00.0-[b1]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:b4]---00.0-[b5]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:b8]---00.0-[b9]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:bc]---00.0-[bd]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:c0]---00.0-[c1]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-[0000:c4]---00.0-[c5]----00.0 Red Hat, Inc. Virtio 1.0 network device
\-[0000:c8]-+-00.0-[c9]----00.0 Red Hat, Inc. Virtio 1.0 network device
+-01.0-[ca]----00.0 Red Hat, Inc. Virtio 1.0 network device
\-02.0-[cb]----00.0 Red Hat, Inc. Virtio 1.0 network device
Thanks,
Nathan
