+1 to consider moving v6 to mandatory. Anyone in a crunch for addresses would agree... Al
From: [email protected] [mailto:[email protected]] On Behalf Of Eddie Arrage Sent: Tuesday, June 26, 2018 1:47 PM To: Georg Kunz <[email protected]>; [email protected] Cc: [email protected] Subject: Re: [opnfv-tech-discuss] [opnfv-tsc] Review of proposed scope for OVP 2018.0x Hi Georg, Thanks for compiling all this information! In additional to the incremental test cases between releases, another consideration should be moving some of the 'optional' test areas from the 2018.01 release to 'mandatory' in this release. It makes a lot of sense to at least consider moving the IPv6 test cases from optional to mandatory. IPv6 should be table stakes at this point - this is my opinion anyway. All the best, Eddie From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Georg Kunz Sent: Tuesday, June 26, 2018 8:25 AM To: [email protected]<mailto:[email protected]> Cc: [email protected]<mailto:[email protected]> Subject: [opnfv-tsc] Review of proposed scope for OVP 2018.0x Dear TSC, I promised during today's TSC call to provide more detailed information on the proposed scope of the next OVP release (2018.0x): * Slides showing a concise mapping of test suites to proposed categories and requirement levels (mandatory / optional) https://wiki.opnfv.org/download/attachments/2925933/Dovetail-OVP%20-%20TSC%20scope%20review.pdf?version=1&modificationDate=1530025340369&api=v2<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_download_attachments_2925933_Dovetail-2DOVP-2520-2D-2520TSC-2520scope-2520review.pdf-3Fversion-3D1-26modificationDate-3D1530025340369-26api-3Dv2&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=1J6hOGYLT1xdJQ1laL6kpfnwJMgz5xGe4OxJgxe-FE8&s=1zJTB7uIiWWsbr4IE97IfWWHxr7n3zdRX8QFQdxsWno&e=> * List of all proposed new test cases to be included in addition to the existing scope (OVP 2018.01) https://wiki.opnfv.org/display/dovetail/Proposed+Test+Scope+of+OVP+2018.0x<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_display_dovetail_Proposed-2BTest-2BScope-2Bof-2BOVP-2B2018.0x&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=1J6hOGYLT1xdJQ1laL6kpfnwJMgz5xGe4OxJgxe-FE8&s=_fDKeR9oi1yBkneoab_9sNA0xzdXQEQf-HgJFtVM9Mw&e=> * State of the proposed test cases in our CI https://wiki.opnfv.org/display/dovetail/Status+of+proposed+tests<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_display_dovetail_Status-2Bof-2Bproposed-2Btests&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=1J6hOGYLT1xdJQ1laL6kpfnwJMgz5xGe4OxJgxe-FE8&s=Qnxg3GBHad2LPX1L8ygwtTFo2rAP3pd2Xx_crBsw8Dw&e=> Please note that some of the failing test cases do in fact pass in the normal daily CI runs of the installers. The dovetail test suite is executed after Functest and Yardstick and we do see an irregular (PASS/FAIL) behavior across all platforms. I attribute this to the fact that the deployments have already gone through quite some test load which causes instabilities. If a test passes regularly in the daily installer runs and shows unstable behavior in the Dovetail runs, we still consider it mature enough for inclusion. * Bottlenecks ping stress test There was a request for more information on the bottlenecks stress test we are proposing for inclusion. In a nutshell, the test spawns a configurable number of threads, each deploying a Heat stack consisting of two VMs through Yardstick. The two VMs attempt to verify connectivity by means of pings. The test passes if the number of successfully deployed stacks is greater than a configurable threshold. In its current configuration, the test deploys 2x 20 stacks / threads and requires that all 20 stacks deploy successfully [1]. Unfortunately, we observe the testcase to fail regularly (observe the "result" and "success_rate" fields: [2]) given its current parameterization. We received the input on the call that Dovetail should not change the parameterization of test cases, but delegate this to the corresponding testing projects. [1] https://github.com/opnfv/bottlenecks/blob/master/testsuites/posca/testcase_cfg/posca_factor_ping.yaml<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_opnfv_bottlenecks_blob_master_testsuites_posca_testcase-5Fcfg_posca-5Ffactor-5Fping.yaml&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=1J6hOGYLT1xdJQ1laL6kpfnwJMgz5xGe4OxJgxe-FE8&s=Xm5xlS43yEak5ps8hbekQLXce2DAU815dJuYo3aPWKg&e=> [2] http://testresults.opnfv.org/test/api/v1/results?project=bottlenecks&case_name=posca_factor_ping<https://urldefense.proofpoint.com/v2/url?u=http-3A__testresults.opnfv.org_test_api_v1_results-3Fproject-3Dbottlenecks-26case-5Fname-3Dposca-5Ffactor-5Fping&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=1J6hOGYLT1xdJQ1laL6kpfnwJMgz5xGe4OxJgxe-FE8&s=oGpXeUKngS89jpwQg_7ztvjXRQ1OEEgSNyQjmzCo06w&e=> Please review the proposal and come back with questions and remarks where needed. We are planning to review and approve the scope in one of the next TSC meetings. Best regards Georg From: Georg Kunz Sent: Thursday, June 14, 2018 10:43 AM To: [email protected]<mailto:[email protected]> Cc: xudan (N) <[email protected]<mailto:[email protected]>>; Cooper, Trevor <[email protected]<mailto:[email protected]>>; Lincoln Lavoie <[email protected]<mailto:[email protected]>> Subject: [opnfv-tsc] Review of proposed scope for OVP 2018.0x Dear TSC, As part of my OVP release update during Tuesday's TSC call, I took an action item to compile a wiki page providing an overview of the new test cases under evaluation for the next OVP release as well as some pointers to in-depth information about the tests (documentation or code): https://wiki.opnfv.org/display/dovetail/Proposed+Test+Scope+of+OVP+2018.0x<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_display_dovetail_Proposed-2BTest-2BScope-2Bof-2BOVP-2B2018.0x&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=1J6hOGYLT1xdJQ1laL6kpfnwJMgz5xGe4OxJgxe-FE8&s=_fDKeR9oi1yBkneoab_9sNA0xzdXQEQf-HgJFtVM9Mw&e=> Please feel free to review and comment. The page might undergo updates as we are finalizing the scope. Best regards Georg -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21469): https://lists.opnfv.org/g/opnfv-tech-discuss/message/21469 Mute This Topic: https://lists.opnfv.org/mt/22685504/21656 Group Owner: [email protected] Unsubscribe: https://lists.opnfv.org/g/opnfv-tech-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
