hi, On Thu, Dec 3, 2015 at 11:19 AM, Armando M. <arma...@gmail.com> wrote: > Hi Neutrinos, > > I would like to share a proposal with you on how we could scale our > ever-growing testing needs, and at the same time, provide guidance to the > developers who care about the quality of the work they produce, and how they > can protect it from the dreadful regressions! > > In [1] you will see a Decision Tree: the objective is to guide us in > deciding what is the best testing strategy for a feature, and what type of > support to expect from our CI engine. In a nutshell we have the following > basic options we can choose from: > > Unit testing > Functional testing > API testing (*) > Fullstack testing > Scenario testing (**) > > The bottom line is: we should tap into our existing infrastructure as much > as we can to try to minimize the amount of CI moving parts that allow us to > test the numerous features that Neutron has. Some work has to happen (as > highlighted below) before we can make this decision process as smooth as > possible. In particular: > > (*) The API testing framework allows us to execute tempest-style API tests > against a Neutron live server. Assaf and I are working to address the > overlap between Tempest and this framework, and to ensure that there's a > clear demarcation to what test belongs to Tempest and what test belongs to > Neutron codebases respectively. More will follow in another thread, so stay > tuned. On the other hand, there's some work that needs to go into this > testing framework to make the server load up non-default extensions. > > (**) This is something that Ihar started in [2]. I am still unclear on how > we intend to leverage this job and yet keep scenario-like tests for hardcore > Neutron features in the Neutron tree. Ihar is thinking QoS, I may be > thinking dpdk, someone else might be thinking to something yet again > different (don't get bogged down on the actual examples). We'll iterate on > [2], but I see it as an integral part of our end-to-end testing strategy. > More to follow. > > As for new additional jobs that may come and go: I believe that we have to > revisit our approach to introduce experimental and non-voting jobs, and how > we make them stable and ultimately running as gate jobs and, long term, > having the features they tests supplant the ones they are meant to supersede > (think pecan, or dvr). I have an idea on how to address the non-voting > neglect conundrum, but I'll tackle it in another thread. > > Finally, a word about projects that need testing with Neutron, advanced > services and beyond. The same way the neutron-lib initiative is tackling the > python side of the Neutron codebase as of now, we'll have to work to > identify the common pieces of functionality that will allow the reuse of the > testing machinery across multiple project, in a modular and decouple > fashion, so that when Neutron related projects want to integrate with the > Neutron CI engine, they will do so using good patterns rather than bad ones; > I believe that [3] is one of them, and that's why I have been pushing back.
can you explain what distinguishes good ones and bad ones? > > Feedback welcome, as I am sure I may overlooked something. > > Cheers, > Armando > > [1] https://wiki.openstack.org/wiki/Network/Testing > [2] https://review.openstack.org/#/c/247697/ > [3] https://review.openstack.org/#/c/242925/ > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
