Hello Alec, I fully agree with you. Yes Functest team would have to work on 3 branches (and it's not only about cherry-picks): - stable/fraser (bugfixes) - stable/gambia (dev) - master (dev) It may be possible only if - Functest team accepts the additionnal work, - OPNFV implements a new CI/CD job design for Test Frameworks (we can't wait for weeks if we have to maintain 3 stable branches) - we fully redefine our objectives (removing dependencies and splitting our repos would become the first steps but it seems much more relevant to verify operating VIM), >From the time being, we can ony support the traditionnal track as all technical difficulties are unmet.Please find below my answers to David and TSC about G: Cédric I said that Functest is forced to follow both tracks else the new OPNFV models can hardly be successful (it may work if OpenStack doesn't introduce big changes but who knows at the beginning of th new OpenStack release?). The "master" model is built from a falsy upward compatibility as we have already discussed during your release meetings. We shouldn't verify scenarios following VIM master by older VIM clients (Gambia) even if it could work by exceptions. Could I please get the latest results of Functest vs XCI?
I'm still thinking about all technical impacts in Functest and OPNFV Features testcases: - requirements synchronization (master could be equal to VIM master or not depending on the OPNFV Features) - dependencies (at least we would have to rewrite our Healthcheck testcases due to the dependency to snaps) - duplicated work (everything has to be cherry-picked or we do skip OpenStack Queen) - possible testcase exclusions (our vnf testcases depend on middlewares such as orchestrators which couldn't support master) - etc. And we can't support 2 stable branches with the current daily job design: - no control loop - wait for days and weeks the results (it will be worse as stable scenarios will run in weekly job) If I must answer right now, we will ony support the traditionnal track. (And we won't hack Functest to verify a nested master installer (i.e. to bypass timeout issues) or to solve possible API version incompatibilities) Lots of key challenges have already been listed and from the time being only Healtcheck is selected as release criteria and as verify step in the new models. https://wiki.opnfv.org/display/SWREL/User+stories+for+OPNFV+release+art ifacts http://testresults.opnfv.org/functest/gambiachallenges/ https://gerrit.opnfv.org/gerrit/#/c/55951/ Tim (Rozet) and Tim (Irnich) have already proposed their helps regarding Functional Gating which is a good start for tracking both. http://testresults.opnfv.org/functest/gates/ My concern about CI/CD optimizations (stable scenarios will run once a week) is opened whatever the tracks supported. On dim., 2018-04-22 at 15:43 +0000, Alec Hothan (ahothan) wrote: > [sorry sent previous email too early] > > > Traditional OpenStack testing tool have had a pretty hard time to > support multiple openstack releases because the openstack API > community has been very opinionated in managing API evolution and > backward > compatibility. > > I’d like to provide some feedback from my perspective, if you forget > for a moment how functest is designed today and testing tools that > must run in functest in order to run at all… > > It is still possible for openstack apps in general to support > multiple openstack releases but that requires constant adjustment and > potentially supporting multiple branches in the worst cases. Whether > a project > can manage to evade multiple branches depends on its footprint to > openstack APIs and some luck. > A simple example is the openstack api to manage floating IPs was > removed starting from python neutron client 10.0.0, following a few > releases of deprecation status, of course that broke a lot of apps > but luckily > there was an easy temporary fix to it: restrict the dependencies to > use a version of that API < 10.0.0 and it was luckily working again. > The better solution would be to change the code to use the new API > for managing floating IPs but that is not an easy path > because such API may not even exist or be properly documented (the > case of floating IP being pulled without proper replacement API with > proper documentation is just one example of miscommunication between > the various teams involved in OpenStack). > Python also allows tools to be able to support multiple APIs at > runtime, so you could for example try something (old API) and if it > fails try the second way – and still manage to avoid multiple > branches. > > So it is not necessarily that bad for all apps to have one branch > (master) supporting multiple openstack releases (e.g. releases in CD > and traditional track), For example nfvbench has managed so far to > support > releases spanning from Liberty to Queens using just master. > However that may not be that easy for every project … Functest in > particular is at the opposite end as it is designed to control the > exact set of library dependencies for every release and in a way that > is > strictly aligned to the openstack release under test. Meaning that > some of the solutions I described above are not available to test > projects that run under Functest: for example, you can’t downgrade > your own project dependency because it is imposed by Functest > (more precisely Functest will build you the container image that > only has the strict set of libraries). But you could still program > your way around it (e.g. at runtime try old way and if it fails try > new way) but that requires proper adjustment in the code. > > Having said that, it will be up to the Barometer team to find out if > they can handle more than 1 openstack release and I do agree it can > potentially be a lot of work doing so. > > (Fatih) Perhaps a useful precision to provide to the community is an > estimate of the spread in openstack release between the traditional > tracks and the CD tracks. From what I can see it should not be more > than 3 openstack releases? > > > Alec > > > > > > > > > From: > <[email protected]> on behalf of "ollivier.c > [email protected]" <[email protected]> > > Date: Saturday, April 21, 2018 at 9:22 AM > > To: David McBride <[email protected]>, Aaron Smith <aasmit > [email protected]> > > Cc: "[email protected]" <[email protected] > nfv.org> > > Subject: Re: [opnfv-tech-discuss] [barometer] > > > > > > Both? Technically very difficult and it duplicates branches and all > patchsets if barometer follows both master and stable. > > > In case of Features integrated in Functest, it's even much more > complex as it raises side effects on Functest, Snaps, requirement > synchronization over the OPNFV projects, > etc. > > > > > > Again the topic is interesting but we may take all technical details > into consideration (see threads about first tests again XCI). > > > From the time being, the model mostly fits a virtual installer and > Apex. > > > > > > From the time being, the full model is built from a falsy upward > compatibility (already discussed during release meeting). > > > > And all dependencies and needs regarding Functest are unmet. > > > > > > I will open a full thread about the > falsy upward compatibility. > > > > > > > Cédric > > > > > > On ven., 2018-04-20 at 15:32 -0700, David McBride wrote: > > > > Do both! :) > > > > > On Fri, Apr 20, 2018 at 1:32 PM, Aaron Smith <[email protected]> > wrote: > > > > > > > I am assuming participation in the Gambia release? > > > > > > The question is whether to switch to XCI or continue with the > traditional release. > > > > > > Feedback? > > > > > > Aaron > > -- > > > Mobile > > > > > _______________________________________________ > > opnfv-tech-discuss mailing list > > [email protected] > > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > > > > > > > > > > > -- > > > > > > > > David McBride > > > Release Manager, OPNFV > > > Mobile: +1.805.276.8018 > > > Email/Google Talk: [email protected] > > > Skype: davidjmcbride1 > > > IRC: dmcbride > > > > > > > > > _______________________________________________ > opnfv-tech-discuss mailing list > [email protected] > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > > > >
_______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
