On Fri, Jun 27, 2025 at 06:39:24PM -0000, mitchell.augus...@canonical.com wrote:
Hi,I'm trying to get a better understanding of how libvirt VMs interact with the default QEMU setting for pci-hole64-size on q35 hosts, to assess why my libvirt VMs behave differently from a similarly configured lxd VM. As I understand, both libvirt and lxd are using qemu q35 VMs under the hood, and both are inheriting their pci-hole64-size from qemu's default setting (correct me if that's wrong), but in my tests, I'm getting different behavior from them. I know lxd is probably out of scope from the libvirt project perspective, so consider this more of a libvirt question w/ some added lxd context. All of this is on a DGX B200 host, which contains large (~180GB VRAM) GPUs. With libvirt/virt-install, I created a q35 virtual machine with CPU host passthrough and 1 or more GPUs passed-through via --host-device. Without additional modifications, this works as expected, and I can initialize the GPU driver in the VM and run nvidia-smi. With lxd (which creates a q35 virtual machine with CPU host passthrough by default), I attached 1 GPU via "lxc config device add passthroughtest gpu gpu pci=1b:00.0". On that machine, the pci-hole64-size is too small by default, since I see these in my dmesg: [ 1.099110] pci 0000:00:01.5: bridge window [mem size 0x6000000000 64bit pref]: can't assign; no space [ 1.120274] pci 0000:00:01.5: bridge window [mem size 0x6000000000 64bit pref]: can't assign; no space [ 1.183281] pci 0000:06:00.0: BAR 2 [mem size 0x4000000000 64bit pref]: can't assign; no space [ 1.186320] pci 0000:06:00.0: BAR 0 [mem size 0x04000000 64bit pref]: can't assign; no space [ 1.189340] pci 0000:06:00.0: BAR 4 [mem size 0x02000000 64bit pref]: can't assign; no space and I cannot initialize the GPU driver since the BARs weren't mapped correctly. When I apply a larger hole size to my lxd VM via `lxc config set passthroughtest raw.qemu=' -global q35-pcihost.pci-hole64-size=8192G'`, I don't see any "can't assign; no space" messages, and the driver works as expected. My question about libvirt is - where (if at all) does libvirt interact with qemu's pci-hole64-size value? If libvirt does not automatically do something functionally similar to changing the hole size like I need to do above for lxd, and is in fact just using a qemu default value, is there some other related interaction happening in libvirt that might explain why my libvirt VMs don't require a manual change to pci-hole64-size, despite the fact that the relevant parts of the underlying qemu machine should be the same?
Unless you set it in the XML we don't do anything with it AFAIK. It might, however be related to something else, like topology. I would suggest you dump the qemu command line for both use cases and ask on qemu ML if you want to get into the differences. Have a nice day, Martin
signature.asc
Description: PGP signature
