On 7/4/2023 8:41 PM, Ding, Xuan wrote:
Hi Nipun,
-----Original Message-----
From: Gupta, Nipun <nipun.gu...@amd.com>
Sent: Tuesday, July 4, 2023 5:23 PM
To: Ding, Xuan <xuan.d...@intel.com>; dev@dpdk.org; tho...@monjalon.net;
Burakov, Anatoly <anatoly.bura...@intel.com>; Yigit, Ferruh
<ferruh.yi...@amd.com>; David Marchand <david.march...@redhat.com>
Cc: Agarwal, Nikhil <nikhil.agar...@amd.com>; He, Xingguang
<xingguang...@intel.com>; Ling, WeiX <weix.l...@intel.com>
Subject: Re: [PATCH] vfio: do not coalesce DMA mappings
Hi Xuan,
On 7/4/2023 1:36 PM, Ding, Xuan wrote:
Hi Nipun,
-----Original Message-----
From: Gupta, Nipun <nipun.gu...@amd.com>
Sent: Tuesday, July 4, 2023 2:54 PM
To: Ding, Xuan <xuan.d...@intel.com>; dev@dpdk.org;
tho...@monjalon.net; Burakov, Anatoly <anatoly.bura...@intel.com>;
Yigit, Ferruh <ferruh.yi...@amd.com>; David Marchand
<david.march...@redhat.com>
Cc: Agarwal, Nikhil <nikhil.agar...@amd.com>; He, Xingguang
<xingguang...@intel.com>; Ling, WeiX <weix.l...@intel.com>
Subject: RE: [PATCH] vfio: do not coalesce DMA mappings
Hi Xuan,
Please see inline.
-----Original Message-----
From: Ding, Xuan <xuan.d...@intel.com>
Sent: Tuesday, July 4, 2023 10:43 AM
To: Gupta, Nipun <nipun.gu...@amd.com>; dev@dpdk.org;
tho...@monjalon.net; Burakov, Anatoly <anatoly.bura...@intel.com>;
Yigit, Ferruh <ferruh.yi...@amd.com>
Cc: Agarwal, Nikhil <nikhil.agar...@amd.com>; He, Xingguang
<xingguang...@intel.com>; Ling, WeiX <weix.l...@intel.com>
Subject: RE: [PATCH] vfio: do not coalesce DMA mappings
Hi Nipun,
-----Original Message-----
From: Ding, Xuan
Sent: Friday, June 30, 2023 1:58 PM
To: Nipun Gupta <nipun.gu...@amd.com>; dev@dpdk.org;
tho...@monjalon.net; Burakov, Anatoly <anatoly.bura...@intel.com>;
ferruh.yi...@amd.com
Cc: nikhil.agar...@amd.com; He, Xingguang <xingguang...@intel.com>;
Ling, WeiX <weix.l...@intel.com>
Subject: RE: [PATCH] vfio: do not coalesce DMA mappings
Hi Nipun,
Replies are inline.
-----Original Message-----
From: Nipun Gupta <nipun.gu...@amd.com>
Sent: Friday, June 30, 2023 9:46 AM
To: Ding, Xuan <xuan.d...@intel.com>; dev@dpdk.org;
tho...@monjalon.net; Burakov, Anatoly <anatoly.bura...@intel.com>;
ferruh.yi...@amd.com
Cc: nikhil.agar...@amd.com; He, Xingguang
<xingguang...@intel.com>; Ling, WeiX <weix.l...@intel.com>
Subject: Re: [PATCH] vfio: do not coalesce DMA mappings
Hi Xuan,
Thanks for pointing out the issue and figuring out the patch which
introduced this. If you have answers to below queries, please let me
know:
Is there any other test cases which tests "--no-huge" which pass?
Yes, there are test cases adding "--no-huge" option to validate 4k
page size in async vhost.
Actually, the page size is decided by front-end, so I think this
case can be removed.
Previously, testpmd can start with "--no-huge" options (not sure if
there are test cases).
Cmd: ./build/app/dpdk-testpmd -l 5-6 -n 4 --no-huge -m 1024 -- -i
Also, if we change the "-m" option to provide lower memory, does
the test pass?
"-m" option is also added and does not work.
When you mention too many pages exceed the capability of IOMMU,
you are referring to HW capability to create multiple pages? Here
it seems in case of 4K page size we need 256K pages which is
limiting the
capacity?
Yes, this is the result of my initial debugging.
The direct impact is that this kind of testpmd cases cannot start now.
If this is expected, I think we can close this defect and ignore
the "--no-
huge"
option when start.
Any insights? Should we just ignore the "--no-huge" option and close
this
defect?
Now we did this as a workaround. Seems no one uses the "--no-huge"
option in testpmd now.
VFIO supports dma_entry_limit as a module parameter, which has a
default value of U16_MAX i.e. 64K, most likely which is limiting
creation of 256K entries for 4K pages here. This can be modified while
inserting vfio module:
modprobe vfio_iommu_type1 dma_entry_limit=1000000
Thanks for your suggestion. I tried it on ubuntu 22.04 but it does not work.
The reason I think is vfio-pci is build-in in kernel driver (since 20.04) and it
does not support dynamic insmod/rmmod.
Does this command need to rmmod vfio first and then modprobe again?
If it is inserted as a module then you can remove using rmmod and then
modprobe again with the dma_entry_limit parameter. Also note,
vfio_iommu_type1 is the module which is limiting the entries to 64K, so this
module needs to be inserted again providing the dma_entry_limit module
param.
In case the module is built-in you can provide via kernel command line
parameter (ref:
https://www.kernel.org/doc/html/v4.12/admin-guide/kernel-
parameters.html).
As per this ref document, "vfio_iommu_type1.dma_entry_limit=1000000"
should be used in the bootargs to set the module parameters.
FYI.. DPDK documentation also mentions the limitation at:
https://doc.dpdk.org/guides/linux_gsg/linux_drivers.html
Thanks for pointing out these references.
Add supplement for configuring build-in vfio module:
Except adding " vfio_iommu_type1.dma_entry_limit=1000000" in bootargs,
we can use kernel command line: "echo 1000000 >
/sys/module/vfio_iommu_type1/parameters/dma_entry_limit".
Both methods works, thanks again.
Good to know this works!
Regards,
Xuan
Thanks,
Nipun
