> > > > Hi, > > > > > > > > The qemu process will stuck when hot-add large size memory to > > > > the virtual machine with a device passtrhough. > > > > We found it is too slow to pin and map pages in vfio_dma_do_map. > > > > Is there any method to improve this process? > > > > > > At what size do you start to see problems? The time to map a > > > section of memory should be directly proportional to the size. As > > > the size is increased, it will take longer, but I don't know why > > > you'd reach a point of not making forward progress. Is it actually > > > stuck or is it just taking longer than you want? Using hugepages > > > can certainly help, we still need to pin each PAGE_SIZE page within > > > the hugepage, but we'll have larger contiguous regions and therefore > > > call iommu_map() less frequently. Please share more data. Thanks, > > > > > > Alex > > It just take longer time, instead of actually stuck. > > We found that the problem exist when we hot-added 16G memory. And it > > will consume tens of minutes when we hot-added 1T memory. > > Is the stall adding 1TB roughly 64 times the stall adding 16GB or do we > have some inflection in the size vs time curve? There is a cost to > pinning an mapping through the IOMMU, perhaps we can improve that, but I > don't see how we can eliminate it or how it wouldn't be at least linear > compared to the size of memory added without moving to a page request > model, which hardly any hardware currently supports. A workaround might > be to incrementally add memory in smaller chunks which generate a less > noticeable stall. Thanks, > > Alex I collected a part of report as below recorded by perf when I hot-added 24GB memory: + 63.41% 0.00% qemu-kvm qemu-kvm-2.8.1-25.127 [.] 0xffffffffffc7534a + 63.41% 0.00% qemu-kvm [kernel.vmlinux] [k] do_vfs_ioctl + 63.41% 0.00% qemu-kvm [kernel.vmlinux] [k] sys_ioctl + 63.41% 0.00% qemu-kvm libc-2.17.so [.] __GI___ioctl + 63.41% 0.00% qemu-kvm qemu-kvm-2.8.1-25.127 [.] 0xffffffffffc71c59 + 63.10% 0.00% qemu-kvm [vfio] [k] vfio_fops_unl_ioctl + 63.10% 0.00% qemu-kvm qemu-kvm-2.8.1-25.127 [.] 0xffffffffffcbbb6a + 63.10% 0.02% qemu-kvm [vfio_iommu_type1] [k] vfio_iommu_type1_ioctl + 60.67% 0.31% qemu-kvm [vfio_iommu_type1] [k] vfio_pin_pages_remote + 60.06% 0.46% qemu-kvm [vfio_iommu_type1] [k] vaddr_get_pfn + 59.61% 0.95% qemu-kvm [kernel.vmlinux] [k] get_user_pages_fast + 54.28% 0.02% qemu-kvm [kernel.vmlinux] [k] get_user_pages_unlocked + 54.24% 0.04% qemu-kvm [kernel.vmlinux] [k] __get_user_pages + 54.13% 0.01% qemu-kvm [kernel.vmlinux] [k] handle_mm_fault + 54.08% 0.03% qemu-kvm [kernel.vmlinux] [k] do_huge_pmd_anonymous_page + 52.09% 52.09% qemu-kvm [kernel.vmlinux] [k] clear_page + 9.42% 0.12% swapper [kernel.vmlinux] [k] cpu_startup_entry + 9.20% 0.00% swapper [kernel.vmlinux] [k] start_secondary + 8.85% 0.02% swapper [kernel.vmlinux] [k] arch_cpu_idle + 8.79% 0.07% swapper [kernel.vmlinux] [k] cpuidle_idle_call + 6.16% 0.29% swapper [kernel.vmlinux] [k] apic_timer_interrupt + 5.73% 0.07% swapper [kernel.vmlinux] [k] smp_apic_timer_interrupt + 4.34% 0.99% qemu-kvm [kernel.vmlinux] [k] gup_pud_range + 3.56% 0.16% swapper [kernel.vmlinux] [k] local_apic_timer_interrupt + 3.32% 0.41% swapper [kernel.vmlinux] [k] hrtimer_interrupt + 3.25% 3.21% qemu-kvm [kernel.vmlinux] [k] gup_huge_pmd + 2.31% 0.01% qemu-kvm [kernel.vmlinux] [k] iommu_map + 2.30% 0.00% qemu-kvm [kernel.vmlinux] [k] intel_iommu_map
It seems that the bottleneck is trying to pin pages through get_user_pages instead of do iommu mapping. Thanks, Wu Zongyong _______________________________________________ vfio-users mailing list vfio-users@redhat.com https://www.redhat.com/mailman/listinfo/vfio-users
