Hi, I’ll only comment on this claim regarding XCI; ”Healtcheck is selected as release criteria and as verify step in the new models.”
As highlighted in the commit message of that change and as reiterated in response to the comment put by Cedric, the purpose of that change was to capture the state of things with XCI at that point in time. New changes are up for collecting feedback from the wider community in order to identify criterias for the entire pipeline. https://lists.opnfv.org/pipermail/opnfv-tech-discuss/2018-April/020991.html /Fatih /Fatih On 22 Apr 2018, at 20:17, [email protected] wrote: Hello Alec, I fully agree with you. Yes Functest team would have to work on 3 branches (and it's not only about cherry-picks): - stable/fraser (bugfixes) - stable/gambia (dev) - master (dev) It may be possible only if - Functest team accepts the additionnal work, - OPNFV implements a new CI/CD job design for Test Frameworks (we can't wait for weeks if we have to maintain 3 stable branches) - we fully redefine our objectives (removing dependencies and splitting our repos would become the first steps but it seems much more relevant to verify operating VIM), From the time being, we can ony support the traditionnal track as all technical difficulties are unmet. Please find below my answers to David and TSC about G: Cédric I said that Functest is forced to follow both tracks else the new OPNFV models can hardly be successful (it may work if OpenStack doesn't introduce big changes but who knows at the beginning of th new OpenStack release?). The "master" model is built from a falsy upward compatibility as we have already discussed during your release meetings. We shouldn't verify scenarios following VIM master by older VIM clients (Gambia) even if it could work by exceptions. Could I please get the latest results of Functest vs XCI? I'm still thinking about all technical impacts in Functest and OPNFV Features testcases: - requirements synchronization (master could be equal to VIM master or not depending on the OPNFV Features) - dependencies (at least we would have to rewrite our Healthcheck testcases due to the dependency to snaps) - duplicated work (everything has to be cherry-picked or we do skip OpenStack Queen) - possible testcase exclusions (our vnf testcases depend on middlewares such as orchestrators which couldn't support master) - etc. And we can't support 2 stable branches with the current daily job design: - no control loop - wait for days and weeks the results (it will be worse as stable scenarios will run in weekly job) If I must answer right now, we will ony support the traditionnal track. (And we won't hack Functest to verify a nested master installer (i.e. to bypass timeout issues) or to solve possible API version incompatibilities) Lots of key challenges have already been listed and from the time being only Healtcheck is selected as release criteria and as verify step in the new models. https://wiki.opnfv.org/display/SWREL/User+stories+for+OPNFV+release+artifacts http://testresults.opnfv.org/functest/gambiachallenges/ https://gerrit.opnfv.org/gerrit/#/c/55951/ Tim (Rozet) and Tim (Irnich) have already proposed their helps regarding Functional Gating which is a good start for tracking both. http://testresults.opnfv.org/functest/gates/ My concern about CI/CD optimizations (stable scenarios will run once a week) is opened whatever the tracks supported. > On dim., 2018-04-22 at 15:43 +0000, Alec Hothan (ahothan) wrote: > [sorry sent previous email too early] > > Traditional OpenStack testing tool have had a pretty hard time to support > multiple openstack releases because the openstack API community has been very > opinionated in managing API evolution and backward compatibility. > I’d like to provide some feedback from my perspective, if you forget for a > moment how functest is designed today and testing tools that must run in > functest in order to run at all… > > It is still possible for openstack apps in general to support multiple > openstack releases but that requires constant adjustment and potentially > supporting multiple branches in the worst cases. Whether a project can manage > to evade multiple branches depends on its footprint to openstack APIs and > some luck. > A simple example is the openstack api to manage floating IPs was removed > starting from python neutron client 10.0.0, following a few releases of > deprecation status, of course that broke a lot of apps but luckily there was > an easy temporary fix to it: restrict the dependencies to use a version of > that API < 10.0.0 and it was luckily working again. The better solution would > be to change the code to use the new API for managing floating IPs but that > is not an easy path because such API may not even exist or be properly > documented (the case of floating IP being pulled without proper replacement > API with proper documentation is just one example of miscommunication between > the various teams involved in OpenStack). > Python also allows tools to be able to support multiple APIs at runtime, so > you could for example try something (old API) and if it fails try the second > way – and still manage to avoid multiple branches. > > So it is not necessarily that bad for all apps to have one branch (master) > supporting multiple openstack releases (e.g. releases in CD and traditional > track), For example nfvbench has managed so far to support releases spanning > from Liberty to Queens using just master. > However that may not be that easy for every project … Functest in particular > is at the opposite end as it is designed to control the exact set of library > dependencies for every release and in a way that is strictly aligned to the > openstack release under test. Meaning that some of the solutions I described > above are not available to test projects that run under Functest: for > example, you can’t downgrade your own project dependency because it is > imposed by Functest (more precisely Functest will build you the container > image that only has the strict set of libraries). But you could still program > your way around it (e.g. at runtime try old way and if it fails try new way) > but that requires proper adjustment in the code. > > Having said that, it will be up to the Barometer team to find out if they can > handle more than 1 openstack release and I do agree it can potentially be a > lot of work doing so. > > (Fatih) Perhaps a useful precision to provide to the community is an estimate > of the spread in openstack release between the traditional tracks and the CD > tracks. From what I can see it should not be more than 3 openstack releases? > > > Alec > > > > > > > > From: <[email protected]> on behalf of > "[email protected]" <[email protected]> > Date: Saturday, April 21, 2018 at 9:22 AM > To: David McBride <[email protected]>, Aaron Smith > <[email protected]> > Cc: "[email protected]" <[email protected]> > Subject: Re: [opnfv-tech-discuss] [barometer] > > Both? Technically very difficult and it duplicates branches and all patchsets > if barometer follows both master and stable. > In case of Features integrated in Functest, it's even much more complex as it > raises side effects on Functest, Snaps, requirement synchronization over the > OPNFV projects, etc. > > Again the topic is interesting but we may take all technical details into > consideration (see threads about first tests again XCI). > From the time being, the model mostly fits a virtual installer and Apex. > > From the time being, the full model is built from a falsy upward > compatibility (already discussed during release meeting). > And all dependencies and needs regarding Functest are unmet. > > I will open a full thread about the falsy upward compatibility. > > Cédric > > On ven., 2018-04-20 at 15:32 -0700, David McBride wrote: > Do both! :) > > On Fri, Apr 20, 2018 at 1:32 PM, Aaron Smith <[email protected]> wrote: > > I am assuming participation in the Gambia release? > > The question is whether to switch to XCI or continue with the traditional > release. > > Feedback? > > Aaron > -- > Mobile > > _______________________________________________ > opnfv-tech-discuss mailing list > [email protected] > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > > > > > -- > David McBride > Release Manager, OPNFV > Mobile: +1.805.276.8018 > Email/Google Talk: [email protected] > Skype: davidjmcbride1 > IRC: dmcbride > _______________________________________________ > opnfv-tech-discuss mailing list > [email protected] > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss _______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
_______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
