> On Feb 15, 2017, at 11:25 AM, Jerin Jacob <jerin.ja...@caviumnetworks.com> > wrote: > > On Wed, Feb 15, 2017 at 02:27:47PM +0000, Shreyansh Jain wrote: >>> -----Original Message----- >>> From: Wiles, Keith [mailto:keith.wi...@intel.com] >>> Sent: Wednesday, February 15, 2017 7:53 PM >>> To: Shreyansh Jain <shreyansh.j...@nxp.com> >>> Cc: Jan Blunck <jblu...@infradead.org>; dev@dpdk.org >>> Subject: Re: [dpdk-dev] [PATCH 3/7] eal: move virtual device probing into a >>> bus >>> >>> >>>> On Feb 15, 2017, at 8:15 AM, Shreyansh Jain <shreyansh.j...@nxp.com> wrote: >>>> >>>> On Wednesday 15 February 2017 07:41 PM, Shreyansh Jain wrote: >>>>> On Wednesday 15 February 2017 03:32 PM, Jan Blunck wrote: >>>>>> This is a refactoring of the virtual device probing which moves into into >>>>>> a proper bus structure. >>>>>> >>>>>> Signed-off-by: Jan Blunck <jblu...@infradead.org> >>>>>> --- >>>>>> lib/librte_eal/common/eal_common_dev.c | 22 ----------------- >>>>>> lib/librte_eal/common/eal_common_vdev.c | 44 >>>>>> +++++++++++++++++++++++++++++++++ >>>>>> 2 files changed, 44 insertions(+), 22 deletions(-) >>>>>> >>>>> >>>>> [...] >>>>> >>>>>> >>>>>> diff --git a/lib/librte_eal/common/eal_common_vdev.c >>>>>> b/lib/librte_eal/common/eal_common_vdev.c >>>>>> index 7d6e54f..523a3d6 100644 >>>>>> --- a/lib/librte_eal/common/eal_common_vdev.c >>>>>> +++ b/lib/librte_eal/common/eal_common_vdev.c >>>>>> @@ -37,8 +37,10 @@ >>>>>> #include <stdint.h> >>>>>> #include <sys/queue.h> >>>>>> >>>>> [...] >>>>> >>>>>> + >>>>>> +static struct rte_bus rte_vdev_bus = { >>>>>> + .scan = vdev_scan, >>>>>> + .probe = vdev_probe, >>>>>> +}; >>>>>> + >>>>>> +RTE_REGISTER_BUS_LATE(virtual, rte_vdev_bus); >>>>>> >>>>> >>>>> Does it matter if VDEV buses are registered before or after other >>>>> buses? Either way, the callbacks would be called in the order specified >>>>> in EAL. >>>>> >>>>> >>>> >>>> Just ignore this comment - I am misunderstood something. >>>> >>>> But another question: Is there specific reason VDEV should be >>> registered/scanned *after* other devices? Is there some specific problem if >>> we do otherwise? (I think this is should be done, but I don't have a >>> specific >>> reason). >>> >>> Does the bonding driver which uses physical devices need to be registered >>> after physical ones? In Pktgen I noticed the vdev after the physical ports >>> and I could not blacklist them as the bonding driver needed them, which >>> caused the bonding ports to have a greater port number. In the case of >>> pktgen >>> the bonding ports were up around 8 or 10 and caused the display to not show >>> the bonding ports. This is really just a usability problem for the developer >>> using Pktgen. I would like to see the vdev devices first, but as long as the >>> drivers (like bonding) are fine with them being first. >> >> Ah, now I remember - there was a patch from Jerin for this. >> Probably he is the best person to comment here. >> (I don't have much insight here). > > commit f4ce209a8ce5f416b61c76cee773bc54749e2048 > Author: Jerin Jacob <jerin.ja...@caviumnetworks.com> > Date: Sun Nov 20 13:30:50 2016 +0530 > > eal: postpone vdev initialization > > Some platform like octeontx may use pci and > vdev based combined device to represent a logical > dpdk functional device.In such case, postponing the > vdev initialization after pci device > initialization will provide the better view of > the pci device resources in the system in > vdev's probe function, and it allows better > functional subsystem registration in vdev probe > function. > > As a bonus, This patch fixes a bond device > initialization use case. > > example command to reproduce the issue: > ./testpmd -c 0x2 --vdev 'eth_bond0,mode=0, > slave=0000:02:00.0,slave=0000:03:00.0' -- > --port-topology=chained > > root cause: > In existing case(vdev initialization and then pci > initialization), creates three Ethernet ports with > following port ids > 0 - Bond device > 1 - PCI device 0 > 2 - PCI devive 1 > > Since testpmd, calls the configure/start on all the ports on > start up,it will translate to following illegal setup sequence
I guess I see this differently, meaning we modified the system to put vdev devices last only because we do not have clean way to startup the system for pdev/vdev devices. The application should be agnostic to the devices being started and the system needs to determine the correct order without a chicken and egg problem. The test-pmd application just starts from 0 to n to initialize devices, which he should be able to do in any order. It is possible the application could initialize the devices (pdev/vdev) in any order, which the current design would break if they tried to init the bonding driver first. What happens if a vdev needs to be initialized before a pdev device? Not saying we need to solve this problem now, but need to figure this out some how. Maybe we need a priority for pdev/vdev devices to determine init order???? > > 1)bond device configure/start > 1.1) pci device0 stop/configure/start > 1.2) pci device1 stop/configure/start > 2)pci device 0 configure(illegal setup case, > as device in start state) > > The fix changes the initialization sequence and > allow initialization in following valid setup order > 1) pcie device 0 configure/start > 2) pcie device 1 configure/start > 3) bond device 2 configure/start > 3.1) pcie device 0/stop/configure/start > 3.2) pcie device 1/stop/configure/start Regards, Keith
