Good discussions. Another area that we need to consider is the availability of good quality test cases in OPNFV. It is one thing to consider if a particular feature itself is "mature', we also need a set of test cases that can adequately evaluate the feature as part of Dovetail. Projects (and testing projects functest/yardstick) should also evaluate the "maturity" of their test suites as one of the criteria.
Morgan, The old thread on percentage of scenarios is not without valid points, but the relevance/significance of scenarios is a disqualifying issue IMO. I'll illustrate with an example, E.g. I looked into all the scenarios in Colorado in the wiki, and filter them with 1) participates in at least one of C-1.0, 2.0, 3.0 2) supports ha 2) supports 2 or more installers If the above hypothetical criteria sounds reasonable, then there are only 6 scenarios remain. One can imagine a case where someone pass all the other scenarios (39 out of total 45 scenarios = 87%) , but not really address what we think as important. This is the state we found ourselves in at Colorado, which, I hope, future releases can evolve/improve. Here are the 6 scenarios I was able to find. I used this table for source (may have inaccuracy in it - I didn’t try to verify): https://wiki.opnfv.org/display/SWREL/Colorado+Scenario+Status os-odl_l3-nofeature-ha Apex,Compass,Fuel os-nosdn-nofeature-ha Apex,Compass,Fuel, Joid os-odl_l2-nofeature-ha Apex,Compass,Fuel, Joid os-onos-nofeature-ha Apex,Compass,Joid os-odl_l2-bgpvpn-ha Apex,Fuel os-onos-sfc-ha Compass,Fuel,Joid Regards, Wenjing -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of [email protected] Sent: Wednesday, January 18, 2017 6:50 PM To: Tapio Tallgren <[email protected]>; Jose Lausuch <[email protected]>; Christopher Price <[email protected]>; [email protected] Subject: Re: [opnfv-tech-discuss] [dovetail]if the l3vpn feature is completed fully in C release Le 18/01/2017 à 11:07, Tapio Tallgren a écrit : > Good topic, I also felt that the criteria were not too clear. > > My interpretation was that if we are testing a feature that should be > in all OPNFV platforms and which is generally available in the > industry, and which does not require a specific installation tool, > then many OPNFV installers would support it. Perhaps even all of them. part of the feature alignment for "mature" features evoked in the discussion on priorities not realistic for Danube, but could be for E and somehow linked to the discussion on scenario refactoring it is a richness to have several installers until a feature is not mature, it makes fully sense to focus on only 1 installer in specific scenario(s) but when the integration is done and available since 1 or 2 OPNFV versions, the feature should be adopted by most of/all the installers in generic scenario(s) it will be useful for certification (and we are back to an old thread... when we say we cannot certify a feature that is not supported by 80% of the scenarios we are releasing...today that is the case of lots of features that are installer dependant) /Morgan > > -Tapio > > > On 01/18/2017 11:38 AM, Jose Lausuch wrote: >> Me neither. If that were the case, that feature Was tested only in >> Fuel during Colorado. >> >> Let's follow up on Friday. >> >> - Jose - >> >> >> -----Original Message----- >> From: [email protected] >> [mailto:[email protected]] On Behalf Of >> Christopher Price >> Sent: Wednesday, January 18, 2017 9:32 AM >> To: Tapio Tallgren; [email protected] >> Subject: Re: [opnfv-tech-discuss] [dovetail]if the l3vpn feature is >> completed fully in C release >> >> Hmm, >> >> I was not aware that “all installers must support” a feature for >> there to be a dovetail suite to validate it. >> Maybe we should review the “qualification criteria” again on Friday’s >> call. >> >> Completely agree that we need to do this in Gerrit. >> >> / chris >> >> On 2017-01-18, 08:59, "Tapio Tallgren" >> <[email protected] on behalf of >> [email protected]> wrote: >> >> On 01/18/2017 12:53 AM, Dave Neary wrote: >> > Hi Hongbo, Jose, >> > >> > As I was saying on the Dovetail calls, I have some concerns >> about moving >> > tests into the Dovetail test suite too early. >> > >> > In the Dovetail test requirements, we have: >> > >> > "* Test cases must pass on OPNFV reference deployments >> > * Tests must not require a specific NFVi platform >> composition or >> > installation tool >> > * Tests must not require unmerged patches to the relevant >> upstream >> > projects" >> > >> > And in the CVP requirements, we have the following section: >> > >> > "The overall CVP compliance verification scope tied to an >> OPNFV release >> > is determined by the Committee. The OPNFV TSC defines and >> maintains the >> > compliance verification procedures and associated tools. The >> scope is >> > constrained to features, capabilities, components, and interfaces >> > included in an OPNFV release that are generally available in the >> > industry (e.g., through adoption by an upstream community)." >> > >> > >> > I wonder if this functionality is sufficiently widely adopted in >> > commercial NFVi and VIM solutions to pass this bar. >> > >> > Thanks, >> > Dave. >> I have no opinion about L3VPN as such, but I read this to >> mean that the >> code should be part of a released upstream projects and that OPNFV >> installers should all support it. >> What would be the best way to discuss these? Currently, the >> test cases >> are on a wiki page which makes it a little difficult to comment >> them. >> Would it make sense to copy the whole test areas and test cases >> wiki >> page to an Etherpad? Or should the whole page be put to gerrit for >> commenting? >> -Tapio >> _______________________________________________ >> opnfv-tech-discuss mailing list >> [email protected] >> https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss >> >> >> _______________________________________________ >> opnfv-tech-discuss mailing list >> [email protected] >> https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > > > _______________________________________________ > opnfv-tech-discuss mailing list > [email protected] > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss -- Morgan Richomme Orange/ IMT/ OLN/ CNC/ NCA/ SINA Network architect for innovative services Future of the Network community member Open source Orange community manager tel. +33 (0) 296 072 106 mob. +33 (0) 637 753 326 [email protected] _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss _______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
