On 7/13/2017 11:55 PM, santosh wrote:
On Thursday 13 July 2017 04:59 PM, Hemant Agrawal wrote:
On 7/11/2017 11:46 AM, Santosh Shukla wrote:
- Moving late bus scanning to up..just after eal_parsing.
- Auto detect iova mapping mode, based on the result of
rte_bus_scan_iommu_class.
Signed-off-by: Santosh Shukla <santosh.shu...@caviumnetworks.com>
Signed-off-by: Jerin Jacob <jerin.ja...@caviumnetworks.com>
---
lib/librte_eal/linuxapp/eal/eal.c | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/lib/librte_eal/linuxapp/eal/eal.c
b/lib/librte_eal/linuxapp/eal/eal.c
index 2546b55e4..7b4dd70de 100644
--- a/lib/librte_eal/linuxapp/eal/eal.c
+++ b/lib/librte_eal/linuxapp/eal/eal.c
@@ -799,6 +799,16 @@ rte_eal_init(int argc, char **argv)
return -1;
}
+ if (rte_bus_scan()) {
+ rte_eal_init_alert("Cannot scan the buses for devices\n");
+ rte_errno = ENODEV;
+ return -1;
+ }
+
The original place of the bus scan was with the following factors:
1. The bus scan requires the VFIO to be enabled atleast in dpaa2 case.
(VFIO code still need cleanup to be support non-pci cleanly). I tried moving it
before bus_scan, this helped in bus scanning.
bus_scan should do scanning, device enumeration, detecting devices and
interface that device bound to, that interface could be VFIO, UIO, UIO_GENERIC
etc..
PCI bus scanning (in eal/) strictly comply to what I mentioned above, thus
aut-detection works gracefully.
However fslmc_bus 'scan' doesn't do device scanning, instead It call vfio
dependent
code which ideally should fall in 'resource mapping' category,. ideally should
happen at bus probe time.
Example:
rte_fslmc_bus_scan()
--> fslmc_vfio_setup_group
--> fslmc_vfio_process_group
So it is doing _setup_ inside scan ops, which in PCI(/vfio-pci) case happens
at probe time (`vfio_setup_device`).
In order to benefit iova auto-detection infrastructure: fslmc bus should
do to two things:
0) fslmc bus scan should look at /sys/bus/platform/drivers/vfio-platform/*
and find out that devices bind to vfio-platform or not, if yes then update kdrv
entry mentioning interface type example VFIO. That-way flsmc bus gets
capability to
inform rte_bus about IOMMU capable interface. Right now, existing implementation
don't have means to inform rte_bus about his devices like pci_bus has!.
vfio_fsl_mc is bit different from pci, we first get the resource
container and then look for resources as children.
In any case, the reworking of the bus is pending since the support for
many other features are being extended for non-pci buses as well in
dpdk e.g. devargs.
It is in my priority list to clean it up for next release.
1) defer the vfio_seup from scan to bus->probe().
This is a good suggestion. This can solve the initialization issue.
2. During SCAN, the bus may allocate memory to devices or for it's own usages.
rte_malloc or mempool is required in cases to support multi-process
environment. (e.g. dpaa2 create dpbp or dpio device memory using the rte_malloc
call).
If bus scanning adheres to device detection or enumeration then
rte_malloc/mempool
not required, Example eal/pci bus scanning.
And in fslmc bus case: if vfio_setup deferred at bus->probe time then
bus->scan won't have memory dependency.
Since none of the other rte library (mempool, memzone, tailq) is available at
this point, it will create significant restriction on the bus scan.
We will prefer if you can re-introduce the "iova_mode" and allow the
application choose, which mode it want to run.
This auto-detect logic may not work for many buses and it is going
to create serious restrictions on the bus_scan code.
fslmc is only bus besides PCI. Auto-detection works gracefully for PCI-bus.
Can you give a try to said proposal?
Ideally vfio-platform code should sit into eal/vfio like eal/vfio-pci is.
Otherwise it will keep creating problems for new generic framework like we're
discussing one.
if said proposal doesn't work for you then I will re-introduce iova-mode as
eal arg, that will override iova mapping mode. But IMHO, eal arg should be
intermediate solution. Once vfio-platform code properly re-factored and merged,
We should remove those eal iova-mode args.
Thanks for digging into the fslmc code. As I said, this is now my
priority item to get the fslmc bus code refactored. We will target
immediately after 17.08 validation.
However till then, we can only support the iova-mode.
+ /* autodetect the iova mapping mode (default is iova_pa) */
+ if (rte_bus_get_iommu_class() == RTE_IOVA_VA)
+ rte_eal_get_configuration()->iova_mode = RTE_IOVA_VA;
+
if (internal_config.no_hugetlbfs == 0 &&
internal_config.process_type != RTE_PROC_SECONDARY &&
internal_config.xen_dom0_support == 0 &&
@@ -896,12 +906,6 @@ rte_eal_init(int argc, char **argv)
return -1;
}
- if (rte_bus_scan()) {
- rte_eal_init_alert("Cannot scan the buses for devices\n");
- rte_errno = ENODEV;
- return -1;
- }
-
RTE_LCORE_FOREACH_SLAVE(i) {
/*
