On Friday 14 July 2017 02:16 PM, santosh wrote: > On Friday 14 July 2017 01:36 PM, Hemant Agrawal wrote: > >> On 7/14/2017 1:25 PM, santosh wrote: >>> On Friday 14 July 2017 01:09 PM, Hemant Agrawal wrote: >>> >>>> On 7/11/2017 11:46 AM, Santosh Shukla wrote: >>>>> Get iommu class of PCI device on the bus and returns preferred iova >>>>> mapping mode for that bus. >>>>> >>>>> Algorithm for iova scheme selection for PCI bus: >>>>> 0. Look for device attached to vfio kdrv and has .drv_flag set >>>>> to RTE_PCI_DRV_NEED_IOVA_VA. >>>>> 1. Look for any device attached to UIO class of driver. >>>>> 2. Check for vfio-noiommu mode enabled. >>>>> >>>>> If 1) & 2) is false and 0) is true then select >>>>> mapping scheme as iova=va. Otherwise use default >>>>> mapping scheme (iova_pa). >>>>> >>>>> Signed-off-by: Santosh Shukla <santosh.shu...@caviumnetworks.com> >>>>> Signed-off-by: Jerin Jacob <jerin.ja...@caviumnetworks.com> >>>>> --- >>>>> v1 --> v2: >>>>> - Removed Linux version check in vfio_noiommu func. Refer [1]. >>>>> - Extending autodetction logic for _iommu_class. >>>>> Refer [2]. >>>>> >>>>> [1] https://www.mail-archive.com/dev@dpdk.org/msg70108.html >>>>> [2] https://www.mail-archive.com/dev@dpdk.org/msg70279.html >>>>> >>>>> lib/librte_eal/linuxapp/eal/eal_pci.c | 66 >>>>> +++++++++++++++++++++++++ >>>>> lib/librte_eal/linuxapp/eal/eal_vfio.c | 19 +++++++ >>>>> lib/librte_eal/linuxapp/eal/eal_vfio.h | 4 ++ >>>>> lib/librte_eal/linuxapp/eal/rte_eal_version.map | 1 + >>>>> 4 files changed, 90 insertions(+) >>>>> >>>>> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci.c >>>>> b/lib/librte_eal/linuxapp/eal/eal_pci.c >>>>> index 7d9e1a99b..573caa000 100644 >>>>> --- a/lib/librte_eal/linuxapp/eal/eal_pci.c >>>>> +++ b/lib/librte_eal/linuxapp/eal/eal_pci.c >>>>> @@ -45,6 +45,7 @@ >>>>> #include "eal_filesystem.h" >>>>> #include "eal_private.h" >>>>> #include "eal_pci_init.h" >>>>> +#include "eal_vfio.h" >>>>> >>>>> /** >>>>> * @file >>>>> @@ -488,6 +489,71 @@ rte_pci_scan(void) >>>>> return -1; >>>>> } >>>>> >>>>> +/* >>>>> + * Any one of the device bound to uio >>>>> + */ >>>>> +static inline int >>>>> +pci_device_bound_uio(void) >>>>> +{ >>>>> + struct rte_pci_device *dev = NULL; >>>>> + >>>>> + FOREACH_DEVICE_ON_PCIBUS(dev) { >>>>> + if (dev->kdrv == RTE_KDRV_IGB_UIO || >>>>> + dev->kdrv == RTE_KDRV_UIO_GENERIC) { >>>>> + return 1; >>>>> + } >>>>> + } >>>>> + return 0; >>>>> +} >>>>> + >>>>> +/* >>>>> + * Any one of the device has iova as va >>>>> + */ >>>>> +static inline int >>>>> +pci_device_has_iova_va(void) >>>>> +{ >>>>> + struct rte_pci_device *dev = NULL; >>>>> + struct rte_pci_driver *drv = NULL; >>>>> + >>>>> + FOREACH_DRIVER_ON_PCIBUS(drv) { >>>>> + if (drv && drv->drv_flags & RTE_PCI_DRV_NEED_IOVA_VA) { >>>>> + FOREACH_DEVICE_ON_PCIBUS(dev) { >>>>> + if (dev->kdrv == RTE_KDRV_VFIO && >>>>> + rte_pci_match(drv, dev)) >>>>> + return 1; >>>>> + } >>>>> + } >>>>> + } >>>>> + return 0; >>>>> +} >>>>> + >>>>> +/* >>>>> + * Get iommu class of PCI devices on the bus. >>>>> + */ >>>>> +enum rte_iova_mode >>>>> +rte_pci_get_iommu_class(void) >>>>> +{ >>>>> + bool is_vfio_noiommu_enabled; >>>>> + bool has_iova_va; >>>>> + bool is_bound_uio; >>>>> + >>>>> + has_iova_va = pci_device_has_iova_va(); >>>>> + is_bound_uio = pci_device_bound_uio(); >>>>> + is_vfio_noiommu_enabled = vfio_noiommu_is_enabled() == 1 ? 1 : 0; >>>>> + >>>>> + if (has_iova_va && !is_bound_uio && !is_vfio_noiommu_enabled) >>>>> + return RTE_IOVA_VA; >>>>> + >>>> PCI is generally present in all platform including dpaa2. >>>> There may not be any device found or available for dpdk usages in such >>>> cases. The PCI bus will still return RTE_IOVA_PA, which will make the >>>> system mode as PA. >>>> >>> That's the expected behavior. And implementation makes sure >>> that PCI_bus return default mode aka _PA if no-pci device found. >>> >>> Isn't code taking care of same? >>> >> I have attached a PCI device to the board. But it is being managed by kernel >> only. >> >> EAL: PCI device 0000:01:00.0 on NUMA socket 0 >> EAL: probe driver: 8086:10d3 net_e1000_em >> EAL: Not managed by a supported kernel driver, skipped >> >> So, there are devices in the PCI list. But none of them is probed or being >> used by dpdk. >> >> > Therefore _pci_get_iommu_class scan result would be _PA, As no device bound > to dpdk. > >>> Let me walk through the code: >>> >>> has_iova_va = 0 (if no pci device then pci_device_has_iov_va() will return >>> 0). >>> >>> And if (has_iova_va & ,,,) will fail therefore rte_pci_get_iommu_class() >>> retuns RTE_IOVA_PA mode. >>> which is default mode. Right? >>> >> This will create issue for the 2nd bus, which is a VA bus. The combined mode >> will becomes '3', so the system mode will be PA. >> > Yes, If both modes detected at two different bus > then policy is to use default iova mapping mode across the buses(which is > _pa). > > Are you operating on two different mode like _pa for PCI-bus and _va for > fslmc bus in dpaa2?
Is vfio kernel infrastructure for dpaa2 allows case like below: 0) Use PCI- vfio(/iommu) mode and map vfio.dma_map to RTE_IOVA_PA AND 1) Use platform/fslmc vfio-platform mode and map vfio.dma_map to RTE_IOVA_VA? Does dpaa2 supports? (Speculating) Lets say if dpaa2 platform supports above case then will you see any issue if both buses using default iova_mapping (_pa), like dpdk pci has currently?
