Hi Konstantin Please move the second thread, I'm feeling you and Gaetan have the same questions.
From: Ananyev, Konstantin, Tuesday, January 23, 2018 3:35 PM > Hi Matan, > > > > > > > Hi Konstantin > > From: Ananyev, Konstantin, Monday, January 22, 2018 10:49 PM > > > Hi Matan, > > > > > > > -----Original Message----- > > > > From: Matan Azrad [mailto:ma...@mellanox.com] > > > > Sent: Monday, January 22, 2018 1:23 PM > > > > To: Ananyev, Konstantin <konstantin.anan...@intel.com>; Gaëtan > > > > Rivet <gaetan.ri...@6wind.com> > > > > Cc: Thomas Monjalon <tho...@monjalon.net>; Wu, Jingjing > > > > <jingjing...@intel.com>; dev@dpdk.org; Neil Horman > > > > <nhor...@tuxdriver.com>; Richardson, Bruce > > > > <bruce.richard...@intel.com> > > > > Subject: RE: [PATCH v3 7/7] app/testpmd: adjust ethdev port > > > > ownership > > > > > > > > > > > > Hi > > > > From: Ananyev, Konstantin [mailto:konstantin.anan...@intel.com] > > > > > Hi lads, > > > > > > > > > > > > > > > > > Hi Matan, > > > > > > > > > > > > On Fri, Jan 19, 2018 at 01:35:10PM +0000, Matan Azrad wrote: > > > > > > > Hi Konstantin > > > > > > > > > > > > > > From: Ananyev, Konstantin, Friday, January 19, 2018 3:09 PM > > > > > > > > > -----Original Message----- > > > > > > > > > From: Matan Azrad [mailto:ma...@mellanox.com] > > > > > > > > > Sent: Friday, January 19, 2018 12:52 PM > > > > > > > > > To: Ananyev, Konstantin <konstantin.anan...@intel.com>; > > > > > > > > > Thomas Monjalon <tho...@monjalon.net>; Gaetan Rivet > > > > > > > > <gaetan.ri...@6wind.com>; > > > > > > > > > Wu, Jingjing <jingjing...@intel.com> > > > > > > > > > Cc: dev@dpdk.org; Neil Horman <nhor...@tuxdriver.com>; > > > > > > > > > Richardson, Bruce <bruce.richard...@intel.com> > > > > > > > > > Subject: RE: [PATCH v3 7/7] app/testpmd: adjust ethdev > > > > > > > > > port ownership > > > > > > > > > > > > > > > > > > Hi Konstantin > > > > > > > > > > > > > > > > > > From: Ananyev, Konstantin, Friday, January 19, 2018 2:38 > > > > > > > > > PM > > > > > > > > > > To: Matan Azrad <ma...@mellanox.com>; Thomas > Monjalon > > > > > > > > > > <tho...@monjalon.net>; Gaetan Rivet > > > > > <gaetan.ri...@6wind.com>; > > > > > > > > Wu, > > > > > > > > > > Jingjing <jingjing...@intel.com> > > > > > > > > > > Cc: dev@dpdk.org; Neil Horman > <nhor...@tuxdriver.com>; > > > > > > > > > > Richardson, Bruce <bruce.richard...@intel.com> > > > > > > > > > > Subject: RE: [PATCH v3 7/7] app/testpmd: adjust ethdev > > > > > > > > > > port ownership > > > > > > > > > > > > > > > > > > > > Hi Matan, > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > From: Matan Azrad [mailto:ma...@mellanox.com] > > > > > > > > > > > Sent: Thursday, January 18, 2018 4:35 PM > > > > > > > > > > > To: Thomas Monjalon <tho...@monjalon.net>; Gaetan > > > > > > > > > > > Rivet <gaetan.ri...@6wind.com>; Wu, Jingjing > > > > > > > > > > > <jingjing...@intel.com> > > > > > > > > > > > Cc: dev@dpdk.org; Neil Horman > > > > > > > > > > > <nhor...@tuxdriver.com>; > > > > > > > > Richardson, > > > > > > > > > > > Bruce <bruce.richard...@intel.com>; Ananyev, > > > > > > > > > > > Konstantin <konstantin.anan...@intel.com> > > > > > > > > > > > Subject: [PATCH v3 7/7] app/testpmd: adjust ethdev > > > > > > > > > > > port ownership > > > > > > > > > > > > > > > > > > > > > > Testpmd should not use ethdev ports which are > > > > > > > > > > > managed by other DPDK entities. > > > > > > > > > > > > > > > > > > > > > > Set Testpmd ownership to each port which is not used > > > > > > > > > > > by other entity and prevent any usage of ethdev > > > > > > > > > > > ports which are not owned by > > > > > > > > Testpmd. > > > > > > > > > > > > > > > > > > > > > > Signed-off-by: Matan Azrad <ma...@mellanox.com> > > > > > > > > > > > --- > > > > > > > > > > > app/test-pmd/cmdline.c | 89 +++++++++++++++++++-- > ---- > > > ---- > > > > > ------- > > > > > > > > ---- > > > > > > > > > > ----- > > > > > > > > > > > app/test-pmd/cmdline_flow.c | 2 +- > > > > > > > > > > > app/test-pmd/config.c | 37 ++++++++++--------- > > > > > > > > > > > app/test-pmd/parameters.c | 4 +- > > > > > > > > > > > app/test-pmd/testpmd.c | 63 > ++++++++++++++++++++---- > > > ---- > > > > > ---- > > > > > > > > > > > app/test-pmd/testpmd.h | 3 ++ > > > > > > > > > > > 6 files changed, 103 insertions(+), 95 deletions(-) > > > > > > > > > > > > > > > > > > > > > > diff --git a/app/test-pmd/cmdline.c > > > > > > > > > > > b/app/test-pmd/cmdline.c index > > > > > > > > > > > 31919ba..6199c64 100644 > > > > > > > > > > > --- a/app/test-pmd/cmdline.c > > > > > > > > > > > +++ b/app/test-pmd/cmdline.c > > > > > > > > > > > @@ -1394,7 +1394,7 @@ struct cmd_config_speed_all { > > > > > > > > > > > &link_speed) < 0) > > > > > > > > > > > return; > > > > > > > > > > > > > > > > > > > > > > - RTE_ETH_FOREACH_DEV(pid) { > > > > > > > > > > > + RTE_ETH_FOREACH_DEV_OWNED_BY(pid, > > > my_owner.id) { > > > > > > > > > > > > > > > > > > > > Why do we need all these changes? > > > > > > > > > > As I understand you changed definition of > > > > > > > > > > RTE_ETH_FOREACH_DEV(), so no testpmd should work ok > > > > > > > > > > default > > > > > (no_owner case). > > > > > > > > > > Am I missing something here? > > > > > > > > > > > > > > > > > > Now, After Gaetan suggestion RTE_ETH_FOREACH_DEV(pid) > > > > > > > > > will iterate > > > > > > > > over all valid and ownerless ports. > > > > > > > > > > > > > > > > Yes. > > > > > > > > > > > > > > > > > Here Testpmd wants to iterate over its owned ports. > > > > > > > > > > > > > > > > Why? Why it can't just iterate over all valid and ownerless > > > > > > > > ports? > > > > > > > > As I understand it would be enough to fix current problems > > > > > > > > and would allow us to avoid any changes in testmpd (which > > > > > > > > I think is a good > > > > > thing). > > > > > > > > > > > > > > Yes, I understand that this big change is very daunted, But > > > > > > > I think the current a lot of bugs in testpmd(regarding port > > > > > > > ownership) even more > > > > > > daunted. > > > > > > > > > > > > > > Look, > > > > > > > Testpmd initiates some of its internal databases depends on > > > > > > > specific port iteration, In some time someone may take > > > > > > > ownership of Testpmd > > > > > ports and testpmd will continue to touch them. > > > > > > > > > > But if someone will take the ownership (assign new owner_id) > > > > > that port will not appear in RTE_ETH_FOREACH_DEV() any more. > > > > > > > > > > > > > Yes, but testpmd sometimes depends on previous iteration using > > > > internal > > > database. > > > > So it uses internal database that was updated by old iteration. > > > > > > That sounds like just a bug in testpmd that need to be fixed, no? > > > > If Testpmd already took ownership for these ports(like I did), it is ok. > > > > Hmm, why not just to fix testpmd, if there is a bug? > As I said all control ops here are done by one thread, so it should be pretty > easy. > Or are you talking about race conditions? > > > > Any particular places where outdated device info is used? > > > > For example, look for the stream management in testpmd(I think I saw it > there). > > Anything particular? > > > > > > > > > If I look back on the fail-safe, its sole purpose is to have > > > > > > seamless hotplug with existing applications. > > > > > > > > > > > > Port ownership is a genericization of some functions > > > > > > introduced by the fail-safe, that could structure DPDK > > > > > > further. It should allow applications to have a seamless > > > > > > integration with subsystems using port ownership. Without this, > port ownership cannot be used. > > > > > > > > > > > > Testpmd should be fixed, but follow the most common design > > > > > > patterns of DPDK applications. Going with port ownership seems > > > > > > like a paradigm shift. > > > > > > > > > > > > > In addition > > > > > > > Using the old iterator in some places in testpmd will cause > > > > > > > a race for run- > > > > > time new ports(can be created by failsafe or any hotplug code): > > > > > > > - testpmd finds an ownerless port(just now created) by the > > > > > > > old iterator and start traffic there, > > > > > > > - failsafe takes ownership of this new port and start traffic > > > > > > > there. > > > > > > > Problem! > > > > > > > > > > Could you shed a bit more light here - it would be race > > > > > condition between whom and whom? > > > > > > > > Sure. > > > > > > > > > As I remember in testpmd all control ops are done within one > > > > > thread (main lcore). > > > > > > > > But other dpdk entity can use another thread, for example: > > > > Failsafe uses the host thread(using alarm callback) to create a > > > > new port and > > > to take ownership of a port. > > > > > > Hm, and you create new ports inside failsafe PMD, right and then set > > > new owner_id for it? > > > > Yes. > > > > > And all this in alarm in interrupt thread? > > > > Yes. > > > > > If so I wonder how you can guarantee that no-one else will set > > > different owner_id between > > > rte_eth_dev_allocate() and rte_eth_dev_owner_set()? > > > > I check it (see failsafe patch to this series - V5). > > Function: fs_bus_init. > > You are talking about that peace of code: > + ret = rte_eth_dev_owner_set(pid, &PRIV(dev)- > >my_owner); > + if (ret) { > + INFO("sub_device %d owner set failed (%s)," > + " will try again later", i, strerror(ret)); > + continue; > > right? > So you just wouldn't include that device into your failsafe device. > But that probably not what user wanted, especially if he bothered to create a > special new low-level device for you. > If that' s the use case, then I think you need to set device ownership at > creation time - inside dev_allocate(). > Again that would avoid such racing conditions inside testpmd. > > > > > > Could you point me to that place (I am not really familiar with > > > familiar with failsafe code)? > > > > > > > > > > > The race: > > > > Testpmd iterates over all ports by the master thread. > > > > Failsafe takes ownership of a port by the host thread and start using > > > > it. > > > > => The two dpdk entities may use the device at same time! > > > > > > Ok, if failsafe really assigns its owner_id(s) to ports that are > > > already in use by the app, then how such scheme supposed to work at > all? > > > > If the app works well (with the new rules) it already took ownership and > failsafe will see it and will wait until the application release it. > > Ok, and why application would need to release it? > How it would know that failsafe device wants to use it now? > Again where is a guarantee that after app released it some other entity > wouldn't grab it for itself? > > > Every dpdk entity should know which port it wants to manage, If 2 > > entities want to manage the same device - it can be ok and port ownership > can synchronize the usage. > > > > Probably, application which will run fail-safe wants to manage only the > > fail- > safe port and therefor to take ownership only for it. > > > > > I.E. application has a port - it assigns some owner_id != 0 to it, > > > then PMD tries to set its owner_id tot the same port. > > > Obviously failsafe's set_owner() will always fail in such case. > > > > > Yes, and will try again after some time. > > Same question again - how app will know that it has to release the port > ownership? > > > > > > From what I hear we need to introduce a concept of 'default owner id'. > > > I.E. when failsafe PMD is created - user assigns some owner_id to it > (default). > > > Then failsafe PMD generates it's own owner_id and assigns it only to > > > the ports whose current owner_id is equal either 0 or 'default' owner_id. > > > > > > > It is a suggestion and we need to think about it more (I'm talking about it > with Gaetan in another thread). > > Actually I think, if we want a generic solution to the generic problem the > current solution is ok. > > From what I heard - every app that wants to use failsafe PMD would require > quite a lot of changes. > It doesn't look ok to me. > > > > > > > > > > > Obeying the new ownership rules can prevent all these races. > > > > > > > > > > When we discussed RFC of owner_id patch, I thought we all agreed > > > that owner_id API shouldn't be mandatory - i.e. existing apps not > > > required to change to work normally with that. > > > > Yes, it is not mandatory if app doesn't use hotplug. > > > > I think with hotplug it is mandatory in the most cases. > > Yes in failsafe you always install this alarm handler, so even if the app > would > have its own way to handle hotplug devices - it would suddenly need to use > this new owner API - even if it doesn't need to. > Why it has to be? > > > > > And it can ease the secondary process model too. > > > > Again, in the generic ownership problem as discussed in RFC: > > Every entity, include app, should know which ports it wants to manage and > to take ownership only for them. > > > > > Though right now it seems that application changes seems necessary, > > > at least to work ok with failsafe PMD. > > > > And for solving the generic problem of ownership.(will defend from future > issues by sure). > > > > > Which makes we wonder was it some sort of misunderstanding or we did > > > we do something wrong here? > > > > Mistakes can be done all the time, but I think we are all understand the big > issue of ownership and how the current solution solves it. > > fail-safe it is just a current example for the problems which are possible > because of the generic ownership issue. > > Honestly that seems too much changes for the app just to make failsafe PMD > work correctly. > IMO - It should be some way to support it without causing changes in each > DPDK application - otherwise something is wrong with the PMD itself. > If let say that ownership model is required to make failsafe PMD to operate - > it should be done in a transparent way to the user. > Probably something like Gaetan suggested in another mail or so. > Konstantin > > > > > Thanks, > > Matan > > > Konstantin > > > > > > > > The only way to attach/detach port with it - invoke testpmd CLI > > > > > "attach/detach" port. > > > > > > > > > > Konstantin > > > > > > > > > > > > > > > > > Testpmd does not handle detection of new port. If it did, > > > > > > testing fail-safe with it would be wrong. > > > > > > > > > > > > At startup, RTE_ETH_FOREACH_DEV already fixed the issue of > > > > > > registering DEFERRED ports. There are still remaining issues > > > > > > regarding this, but I think they should be fixed. The > > > > > > architecture does not need to be completely moved to port > ownership. > > > > > > > > > > > > If anything, this should serve as a test for your API with > > > > > > common applications. I think you'd prefer to know and debug > > > > > > with testpmd instead of firing up VPP or something like that > > > > > > to determine what went wrong with using the fail-safe. > > > > > > > > > > > > > > > > > > > > In addition > > > > > > > As a good example for well-done application (free from > > > > > > > ownership > > > > > > > bugs) I tried here to adjust Tespmd to the new rules and BTW > > > > > > > to fix a > > > > > > lot of bugs. > > > > > > > > > > > > Testpmd has too much cruft, it won't ever be a good example of > > > > > > a well-done application. > > > > > > > > > > > > If you want to demonstrate ownership, I think you should start > > > > > > an example application demonstrating race conditions and their > mitigation. > > > > > > > > > > > > I think that would be interesting for many DPDK users. > > > > > > > > > > > > > > > > > > > > > > > > > > > So actually applications which are not aware to the port > > > > > > > ownership still are exposed to races, but if there use the > > > > > > > old iterator(with the new > > > > > > change) the amount of races decreases. > > > > > > > > > > > > > > Thanks, Matan. > > > > > > > > Konstantin > > > > > > > > > > > > > > > > > > > > > > > > > > I added to Testpmd ability to take an ownership of ports > > > > > > > > > as the new ownership and synchronization rules > > > > > > > > > suggested, Since Tespmd is a DPDK entity which wants > > > > > > > > > that no one will touch its owned ports, It must allocate > > > > > > > > an unique ID, set owner for its ports (see in main > > > > > > > > function) and recognizes them by its owner ID. > > > > > > > > > > > > > > > > > > > Konstantin > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > -- > > > > > > Gaëtan Rivet > > > > > > 6WIND
