Hi Nipun, > -----Original Message----- > From: Gupta, Nipun <nipun.gu...@amd.com> > Sent: Tuesday, July 4, 2023 5:23 PM > To: Ding, Xuan <xuan.d...@intel.com>; dev@dpdk.org; tho...@monjalon.net; > Burakov, Anatoly <anatoly.bura...@intel.com>; Yigit, Ferruh > <ferruh.yi...@amd.com>; David Marchand <david.march...@redhat.com> > Cc: Agarwal, Nikhil <nikhil.agar...@amd.com>; He, Xingguang > <xingguang...@intel.com>; Ling, WeiX <weix.l...@intel.com> > Subject: Re: [PATCH] vfio: do not coalesce DMA mappings > > Hi Xuan, > > On 7/4/2023 1:36 PM, Ding, Xuan wrote: > > Hi Nipun, > > > >> -----Original Message----- > >> From: Gupta, Nipun <nipun.gu...@amd.com> > >> Sent: Tuesday, July 4, 2023 2:54 PM > >> To: Ding, Xuan <xuan.d...@intel.com>; dev@dpdk.org; > >> tho...@monjalon.net; Burakov, Anatoly <anatoly.bura...@intel.com>; > >> Yigit, Ferruh <ferruh.yi...@amd.com>; David Marchand > >> <david.march...@redhat.com> > >> Cc: Agarwal, Nikhil <nikhil.agar...@amd.com>; He, Xingguang > >> <xingguang...@intel.com>; Ling, WeiX <weix.l...@intel.com> > >> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings > >> > >> Hi Xuan, > >> > >> Please see inline. > >> > >>> -----Original Message----- > >>> From: Ding, Xuan <xuan.d...@intel.com> > >>> Sent: Tuesday, July 4, 2023 10:43 AM > >>> To: Gupta, Nipun <nipun.gu...@amd.com>; dev@dpdk.org; > >>> tho...@monjalon.net; Burakov, Anatoly <anatoly.bura...@intel.com>; > >>> Yigit, Ferruh <ferruh.yi...@amd.com> > >>> Cc: Agarwal, Nikhil <nikhil.agar...@amd.com>; He, Xingguang > >>> <xingguang...@intel.com>; Ling, WeiX <weix.l...@intel.com> > >>> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings > >>> > >>> Hi Nipun, > >>> > >>>> -----Original Message----- > >>>> From: Ding, Xuan > >>>> Sent: Friday, June 30, 2023 1:58 PM > >>>> To: Nipun Gupta <nipun.gu...@amd.com>; dev@dpdk.org; > >>>> tho...@monjalon.net; Burakov, Anatoly <anatoly.bura...@intel.com>; > >>>> ferruh.yi...@amd.com > >>>> Cc: nikhil.agar...@amd.com; He, Xingguang <xingguang...@intel.com>; > >>>> Ling, WeiX <weix.l...@intel.com> > >>>> Subject: RE: [PATCH] vfio: do not coalesce DMA mappings > >>>> > >>>> Hi Nipun, > >>>> > >>>> Replies are inline. > >>>> > >>>>> -----Original Message----- > >>>>> From: Nipun Gupta <nipun.gu...@amd.com> > >>>>> Sent: Friday, June 30, 2023 9:46 AM > >>>>> To: Ding, Xuan <xuan.d...@intel.com>; dev@dpdk.org; > >>>>> tho...@monjalon.net; Burakov, Anatoly <anatoly.bura...@intel.com>; > >>>>> ferruh.yi...@amd.com > >>>>> Cc: nikhil.agar...@amd.com; He, Xingguang > >>>>> <xingguang...@intel.com>; Ling, WeiX <weix.l...@intel.com> > >>>>> Subject: Re: [PATCH] vfio: do not coalesce DMA mappings > >>>>> > >>>>> Hi Xuan, > >>>>> > >>>>> Thanks for pointing out the issue and figuring out the patch which > >>>>> introduced this. If you have answers to below queries, please let me > know: > >>>>> > >>>>> Is there any other test cases which tests "--no-huge" which pass? > >>>> > >>>> Yes, there are test cases adding "--no-huge" option to validate 4k > >>>> page size in async vhost. > >>>> Actually, the page size is decided by front-end, so I think this > >>>> case can be removed. > >>>> > >>>> Previously, testpmd can start with "--no-huge" options (not sure if > >>>> there are test cases). > >>>> Cmd: ./build/app/dpdk-testpmd -l 5-6 -n 4 --no-huge -m 1024 -- -i > >>>> > >>>>> > >>>>> Also, if we change the "-m" option to provide lower memory, does > >>>>> the test pass? > >>>> > >>>> "-m" option is also added and does not work. > >>>> > >>>>> > >>>>> When you mention too many pages exceed the capability of IOMMU, > >>>>> you are referring to HW capability to create multiple pages? Here > >>>>> it seems in case of 4K page size we need 256K pages which is > >>>>> limiting the > >> capacity? > >>>> > >>>> Yes, this is the result of my initial debugging. > >>>> The direct impact is that this kind of testpmd cases cannot start now. > >>>> If this is expected, I think we can close this defect and ignore > >>>> the "--no- > >> huge" > >>>> option when start. > >>> > >>> Any insights? Should we just ignore the "--no-huge" option and close > >>> this > >> defect? > >>> Now we did this as a workaround. Seems no one uses the "--no-huge" > >>> option in testpmd now. > >> > >> VFIO supports dma_entry_limit as a module parameter, which has a > >> default value of U16_MAX i.e. 64K, most likely which is limiting > >> creation of 256K entries for 4K pages here. This can be modified while > inserting vfio module: > >> modprobe vfio_iommu_type1 dma_entry_limit=1000000 > > > > Thanks for your suggestion. I tried it on ubuntu 22.04 but it does not work. > > The reason I think is vfio-pci is build-in in kernel driver (since 20.04) > > and it > does not support dynamic insmod/rmmod. > > > > Does this command need to rmmod vfio first and then modprobe again? > > > > If it is inserted as a module then you can remove using rmmod and then > modprobe again with the dma_entry_limit parameter. Also note, > vfio_iommu_type1 is the module which is limiting the entries to 64K, so this > module needs to be inserted again providing the dma_entry_limit module > param. > > In case the module is built-in you can provide via kernel command line > parameter (ref: > https://www.kernel.org/doc/html/v4.12/admin-guide/kernel- > parameters.html). > As per this ref document, "vfio_iommu_type1.dma_entry_limit=1000000" > should be used in the bootargs to set the module parameters. > > FYI.. DPDK documentation also mentions the limitation at: > https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html
Thanks for pointing out these references. Add supplement for configuring build-in vfio module: Except adding " vfio_iommu_type1.dma_entry_limit=1000000" in bootargs, we can use kernel command line: "echo 1000000 > /sys/module/vfio_iommu_type1/parameters/dma_entry_limit". Both methods works, thanks again. Regards, Xuan > > Thanks, > Nipun
