Is there something special about the pinning step? When I start a new VM with 16G in dedicated hugepages my system becomes quite unresponsive for several seconds, significant packet loss and random hang device oopses if I use preemptive kernel?
On Mon, Nov 21, 2022 at 12:54 PM Alex Williamson <alex.william...@redhat.com> wrote: > > On Sun, 20 Nov 2022 16:36:58 -0800 > Bryan Angelo <bang...@gmail.com> wrote: > > > When passing-through via vfio-pci using QEMU 7.1.0 and OVMF, it appears > > that qemu preallocates all guest system memory. > > > > qemu-system-x86_64 \ > > -no-user-config \ > > -nodefaults \ > > -nographic \ > > -rtc base=utc \ > > -boot strict=on \ > > -machine pc,accel=kvm,dump-guest-core=off \ > > -cpu host,migratable=off \ > > -smp 8 \ > > -m size=8G \ > > -overcommit mem-lock=off \ > > -device vfio-pci,host=03:00.0 \ > > ... > > > > PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND > > 4151 root 20 0 13560.8m *8310.8m* 100.0 52.6 0:25.06 S > > qemu-system-x86_64 > > > > > > If I remove just the vfio-pci device argument, it appears that qemu no > > longer preallocates all guest system memory. > > > > PID USER PR NI VIRT RES %CPU %MEM TIME+ S COMMAND > > 5049 root 20 0 13414.0m *762.4m* 0.0 4.8 0:27.06 S > > qemu-system-x86_64 > > > > > > I am curious if anyone has any context on or experience with this > > functionality. Does anyone know if preallocation is a requirement for VFIO > > with QEMU or if preallocation can be disabled? > > > > I am speculating that QEMU is actually preallocating as opposed to the > > guest touching every page of system memory. > > > This is a necessary artifact of device assignment currently. Any memory > that can potentially be a DMA target for the assigned device needs to be > pinned in the host. By default, all guest memory is potentially a DMA > target, therefore all of guest memory is pinned. A vIOMMU in the guest > can reduce the memory footprint, but the guest will still initially pin > all memory as the vIOMMU is disabled at guest boot/reboot, but this > also trades VM memory footprint for latency, as dynamic mappings > through a vIOMMU to the host IOMMU is a long path. > > Eventually, devices supporting Page Request Interface capabilities can > help to alleviate this, by essentially faulting DMA pages, much like > the processor does for memory. Support for this likely requires new > hardware and software though. Thanks, > > Alex > > _______________________________________________ > vfio-users mailing list > vfio-users@redhat.com > https://listman.redhat.com/mailman/listinfo/vfio-users > _______________________________________________ vfio-users mailing list vfio-users@redhat.com https://listman.redhat.com/mailman/listinfo/vfio-users
