On 23-Jul-19 6:27 AM, Jerin Jacob Kollanukkaran wrote:
-----Original Message-----
From: Stojaczyk, Dariusz <dariusz.stojac...@intel.com>
Sent: Tuesday, July 23, 2019 10:24 AM
To: Jerin Jacob Kollanukkaran <jer...@marvell.com>; Thomas Monjalon
<tho...@monjalon.net>; David Marchand <david.march...@redhat.com>;
Burakov, Anatoly <anatoly.bura...@intel.com>
Cc: dev@dpdk.org
Subject: [EXT] RE: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode selection
-----Original Message-----
From: Jerin Jacob Kollanukkaran [mailto:jer...@marvell.com]
Sent: Tuesday, July 23, 2019 6:19 AM
-----Original Message-----
From: Stojaczyk, Dariusz <dariusz.stojac...@intel.com>
Sent: Tuesday, July 23, 2019 9:06 AM
To: Thomas Monjalon <tho...@monjalon.net>; David Marchand
<david.march...@redhat.com>; Burakov, Anatoly
<anatoly.bura...@intel.com>; Jerin Jacob Kollanukkaran
<jer...@marvell.com>
Cc: dev@dpdk.org
Subject: [EXT] RE: [dpdk-dev] [PATCH v4 0/4] Fixes on IOVA mode
selection
This introduces a regression where uio-bound devies are attached to
a
DPDK
app at runtime.
Just to understand the requirements;
# Is this requirement for SPDK?
# Is brand new PCI device scanned and attached to DPDK at runtime?
# Any specific reason for using uio vs vfio?
Jerin,
Stojaczyk,
There reason to choose VA incase if bus detects DC is following:
- All drivers are expected to work in RTE_IOVA_VA mode, irrespective of
physical address availability.
- By default, the mempool, first asks for IOVA-contiguous memory using
``RTE_MEMZONE_IOVA_CONTIG``. This is slow in RTE_IOVA_PA mode and it may
affect the application boot time.
- It is easy to enable large amount of IOVA-contiguous memory use-cases
with IOVA in VA mode.
It came up in SPDK tests, but it's certainly nothing SPDK-specific, I can't give
you the steps but it should be reproducible even with testpmd.
The PCI device could have been simply hotplugged to the system after DPDK
app start. DPDK didn't know about it at initialization, so it picked
RTE_IOVA_VA and then would fail to attach any UIO-bound device ever
after:
EAL: Expecting 'PA' IOVA mode but current mode is 'VA', not initializing
We have RTE_PCI_DRV_NEED_IOVA_AS_VA devices in DPDK, Which can work
Only on VA. If we default 'PA' incase of DC, then what do with hotplugging on
those devices?
EAL: Driver cannot attach the device (0000:00:09.0)
EAL: Failed to attach device on primary process
UIO is commonly used on systems without IOMMU- including VMs.
The latest machines has IOMMU. Which machines you are testing against,
Can we detect the machines without IOMMU and switch to PA?
A machine without an IOMMU shouldn't have picked IOVA as VA in the first
place. Perhaps this is something we could fix? I'm not sure how to
detected that condition though, i don't think there's a mechanism to
know that for sure. Some kernels create a "iommu" sysfs directories, but
i'm not too sure if they're 1) there for older kernels we support, and
2) always there.
On machines with IOMMU, VFIO should be the default, and we should
discourage people from using igb_uio. Is there any reason why SPDK is
not using VFIO by default?
On my machine, "/sys/devices/virtual/iommu" exists when IOMMU is
enabled, but doesn't exist if it isn't ("/sys/class/iommu" exists in
both cases, but is empty when IOMMU is disabled). Perhaps we could go
off that?
--
Thanks,
Anatoly
