Hi Cedric, Inline…
From: "[email protected]" <[email protected]> Date: Sunday, April 22, 2018 at 11:17 AM To: "Alec Hothan (ahothan)" <[email protected]>, David McBride <[email protected]>, Aaron Smith <[email protected]>, Fatih Degirmenci <[email protected]> Cc: "[email protected]" <[email protected]> Subject: Re: [opnfv-tech-discuss] [barometer] Hello Alec, I fully agree with you. Yes Functest team would have to work on 3 branches (and it's not only about cherry-picks): - stable/fraser (bugfixes) - stable/gambia (dev) - master (dev) Just to be clear when you say “Functest team would have to work on 3 branches”, does it only include functest core code? Or is it functest core + all the test projects that run in Functest? Since Fuctest-core pretty much focuses on orchestrating test tools and retrieving results from them or providing APIs for test tools to report results back to functest core. I would think that openstack dependencies used by functest core itself would be minimal and relatively stable. Isn’t that the case? As for managing multiple branches, if it becomes unavoidable, it is clearly more work but it is not the end of the world and pretty straightforward with git if managed properly. It may be possible only if - Functest team accepts the additionnal work, - OPNFV implements a new CI/CD job design for Test Frameworks (we can't wait for weeks if we have to maintain 3 stable branches) This definitely will need to be addressed by finding a dedicated pod for validating test projects so that new versions can be promoted to gate the 2 tracks. - we fully redefine our objectives (removing dependencies and splitting our repos would become the first steps but it seems much more relevant to verify operating VIM), It is more scalable to have each test project team manage compatibility across the release window (which could be not much work for most) than trying to have functest team do it for every test project that uses functest-core. If we manage to have that regression pod for testing projects in place, then it should not be too hard to verify how each test project fares against a limited set of openstack releases (only those that opnfv needs to support in both tracks). I said that Functest is forced to follow both tracks else the new OPNFV models can hardly be successful (it may work if OpenStack doesn't introduce big changes but who knows at the beginning of th new OpenStack release?). The "master" model is built from a falsy upward compatibility as we have already discussed during your release meetings. We shouldn't verify scenarios following VIM master by older VIM clients (Gambia) even if it could work by exceptions. Could I please get the latest results of Functest vs XCI? I'm still thinking about all technical impacts in Functest and OPNFV Features testcases: - requirements synchronization (master could be equal to VIM master or not depending on the OPNFV Features) - dependencies (at least we would have to rewrite our Healthcheck testcases due to the dependency to snaps) - duplicated work (everything has to be cherry-picked or we do skip OpenStack Queen) - possible testcase exclusions (our vnf testcases depend on middlewares such as orchestrators which couldn't support master) - etc. And we can't support 2 stable branches with the current daily job design: - no control loop - wait for days and weeks the results (it will be worse as stable scenarios will run in weekly job) If I must answer right now, we will ony support the traditionnal track. (And we won't hack Functest to verify a nested master installer (i.e. to bypass timeout issues) or to solve possible API version incompatibilities) Lots of key challenges have already been listed and from the time being only Healtcheck is selected as release criteria and as verify step in the new models. https://wiki.opnfv.org/display/SWREL/User+stories+for+OPNFV+release+artifacts http://testresults.opnfv.org/functest/gambiachallenges/ https://gerrit.opnfv.org/gerrit/#/c/55951/ Tim (Rozet) and Tim (Irnich) have already proposed their helps regarding Functional Gating which is a good start for tracking both. http://testresults.opnfv.org/functest/gates/ My concern about CI/CD optimizations (stable scenarios will run once a week) is opened whatever the tracks supported. We need to see with Fatih how new versions of a test project get pushed to the 2 CI Flows corresponding to the 2 tracks. I think the test community should aim for a workflow where a test project would mainly develop and test itself using the dedicated test pod, then provide stable tags to the CD and/or traditional track (that can run a slightly different release of openstack) for consumption by installers and feature projects. Thanks Alec
_______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
