Hi Jose, Scenario owners express intent-to-release on the scenario status page <https://wiki.opnfv.org/display/SWREL/Euphrates+Scenario+Status>. Scenario owners indicated that they wanted to release each scenario in the recommended list.
I would very much like to use test results to gate release of scenarios. That would be my preference. However, each time this issue has come up, I've been told (a) the test frameworks are not sufficiently mature to be used for gating release; and (b) the community prefers to leave the decision about quality to the scenario owners. So, we are left with (1) deploy status; and (2) scenario owner judgment for determining scenario release. We can certainly change this for future releases. In order to do that, I'd like to have a recommendation on test requirements from the test working group, including which tests are gating, minimum test runs, the number of consecutive iterations that must pass, etc. Once we have approval from the TSC, then we can put those requirements in place. David On Thu, Oct 19, 2017 at 8:01 AM, Jose Lausuch <[email protected]> wrote: > Hi, > > I just want to raise a concern the Functest team has been discussing this > morning. > > This list is maybe good from a deployment prospective, but it seems we are > lowering the bar the older OPNFV gets… I remember in the Arno and > Brahmaputra days the release criteria was to pass Functest/Yardstick, then > we said that scenario owners can decide to release or not release according > to the results. Now it seems that a deployable scenario is eligible to be > released without taking into account the quality, which can be acceptable.. > but the community and the users should know that some of these scenarios do > not even pass Healthcheck or vPing whereas others get 100% success from > Functest. > > I am not saying we should not release them but if I were an user, I would > like to know some details about the quality of the scenarios. I would not > like to deploy something that cannot even spawn VMs or do a simple ping > between them… Maybe for the next time, we could have a general release > document/report about the quality of each scenario and mention the known > issues for those which have problems to pass the minimum set of tests. > Functest is doing this by taking a snapshot of the dashboard at the release > date [1], but I do not think we should be the project responsible for > offering that information, it should be more visible and centralized. > > Regards, > Jose > > [1] http://testresults.opnfv.org/functest/euphrates/ > > > > On 17 Oct 2017, at 22:50, David McBride <[email protected]> > wrote: > > Team, > > Based on scenario owner intent-to-participate, as well as a review of > deploy results, I'm recommending that the following scenarios be released > for Euphrates 5.0: > > *Scenario* *Installer* > os-nosdn-nofeature-ha Apex > os-nosdn-calipso-noha Apex > os-nosdn-nofeature-noha Apex > os-odl-nofeature-ha Apex > os-odl-nofeature-noha Apex > os-nosdn-ovs_dpdk-ha Apex > os-nosdn-ovs_dpdk-noha Apex > os-odl-fdio-noha Apex > os-odl-fdio-ha Apex > os-odl-fdio_dvr-noha Apex > os-nosdn-fdio-noha Apex > os-nosdn-fdio-ha Apex > os-nosdn-kvm_ovs_dpdk-ha Apex > os-nosdn-kvm_ovs_dpdk-noha Apex > os-nosdn-bar-ha Apex > os-nosdn-bar-noha Apex > os-odl-bgpvpn-ha Apex > os-odl-bgpvpn-noha Apex > os-odl-sfc-noha Apex > os-ovn-nofeature-noha Apex > os-odl_l2-moon-ha Compass > os-odl_l2-moon-noha Compass > os-nosdn-nofeature-ha Compass4NFV > os-nosdn-nofeature-noha Compass4NFV > os-odl-sfc-ha Compass4NFV > os-odl-sfc-noha Compass4NFV > k8s-nosdn-nofeature-ha Compass4NFV > os-nosdn-nofeature-ha Daisy > os-odl-nofeature-ha Daisy > os-nosdn-nofeature-noha Fuel/MCP > os-nosdn-nofeature-ha Fuel/MCP > os-nosdn-ovs-noha Fuel/MCP > os-nosdn-ovs-ha Fuel/MCP > os-odl-nofeature-noha Fuel/MCP > os-odl-nofeature-ha Fuel/MCP > k8-nosdn-lb-noha JOID > os-nosdn-nofeature-ha JOID > os-nosdn-lxd-ha JOID > os-nosdn-lxd-noha JOID > os-nosdn-nofeature-noha JOID > os-ocl-nofeature-ha JOID > os-ocl-nofeature-noha JOID > os-nosdn-openbaton-ha JOID > k8-ovn-lb-noha JOID > os-nosdn-nofeature-ha Fuel/MCP (aarch64) > os-odl-nofeature-ha Fuel/MCP (aarch64) > > -- > *David McBride* > Release Manager, OPNFV > Mobile: +1.805.276.8018 > Email/Google Talk: [email protected] > Skype: davidjmcbride1 > IRC: dmcbride > _______________________________________________ > opnfv-tech-discuss mailing list > [email protected] > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > > > -- *David McBride* Release Manager, OPNFV Mobile: +1.805.276.8018 Email/Google Talk: [email protected] Skype: davidjmcbride1 IRC: dmcbride
_______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
