Hello,
I am trying to use qemu to run ROCm in a virtual machine via GPU
passthrough with vfio-pci. While this mostly works out of the box, ROCm
requires PCIe atomics to function properly, and it seems like that this
is a feature that is missing in qemu. The setup is using a simple PCie
topology, where the GPU is connected to a pcie-root-port as follows:
-device pcie-root-port,id=pcie.1
-device vfio-pci,host=<host_pci_address>,bus=pcie.1
When the amdgpu kernel module is loaded in the guest, enabling PCIe
atomics fails because it requires that PCIe root ports support 32- and
64-bit atomic completion, see commits 430a2368 and 8e6d0b69 in Linux.
I patched the required flags into the DevCap2 register in qemu and
everything seems to work fine. I'm interested in getting this change
into qemu proper, but I have some questions about how to do that:
1. In order for PCIe atomics to be supported, every bridge in the path
between the root complex and the device in question needs to support the
routing capability, as well as the device being required to support the
atomic requester/completer capabilities of course. When a certain device
from the host is connected to a root port in qemu, that path will be
abstracted away into the virtual PCIe topology. Should this property
from the host be reflected into the virtual path in qemu, or is it
better to gate this behind a property of the pcie-root-port? For
example, the user could then simply pass -device
pcie-root-port,id=pcie.1,atomics=on to make the pcie-root-port report
that it supports these operations. If this should be reflected from the
host automatically, how should this be tested for? I checked around the
source for how for example the link speed is set, but it seems like that
is not reflected from the host system automatically either.
2. Is anything else required to support PCIe atomics, or did I just get
lucky that my programs work?
Thanks,
Robin Voetter, Stream HPC
