It would be better if we can document the scenarios / cases where individual tests make sense. As a developer this confuses me the most.
On Thu, Dec 3, 2015 at 10:21 AM, Takashi Yamamoto <yamam...@midokura.com> wrote: > hi, > > On Thu, Dec 3, 2015 at 11:19 AM, Armando M. <arma...@gmail.com> wrote: > > Hi Neutrinos, > > > > I would like to share a proposal with you on how we could scale our > > ever-growing testing needs, and at the same time, provide guidance to the > > developers who care about the quality of the work they produce, and how > they > > can protect it from the dreadful regressions! > > > > In [1] you will see a Decision Tree: the objective is to guide us in > > deciding what is the best testing strategy for a feature, and what type > of > > support to expect from our CI engine. In a nutshell we have the following > > basic options we can choose from: > > > > Unit testing > > Functional testing > > API testing (*) > > Fullstack testing > > Scenario testing (**) > > > > The bottom line is: we should tap into our existing infrastructure as > much > > as we can to try to minimize the amount of CI moving parts that allow us > to > > test the numerous features that Neutron has. Some work has to happen (as > > highlighted below) before we can make this decision process as smooth as > > possible. In particular: > > > > (*) The API testing framework allows us to execute tempest-style API > tests > > against a Neutron live server. Assaf and I are working to address the > > overlap between Tempest and this framework, and to ensure that there's a > > clear demarcation to what test belongs to Tempest and what test belongs > to > > Neutron codebases respectively. More will follow in another thread, so > stay > > tuned. On the other hand, there's some work that needs to go into this > > testing framework to make the server load up non-default extensions. > > > > (**) This is something that Ihar started in [2]. I am still unclear on > how > > we intend to leverage this job and yet keep scenario-like tests for > hardcore > > Neutron features in the Neutron tree. Ihar is thinking QoS, I may be > > thinking dpdk, someone else might be thinking to something yet again > > different (don't get bogged down on the actual examples). We'll iterate > on > > [2], but I see it as an integral part of our end-to-end testing strategy. > > More to follow. > > > > As for new additional jobs that may come and go: I believe that we have > to > > revisit our approach to introduce experimental and non-voting jobs, and > how > > we make them stable and ultimately running as gate jobs and, long term, > > having the features they tests supplant the ones they are meant to > supersede > > (think pecan, or dvr). I have an idea on how to address the non-voting > > neglect conundrum, but I'll tackle it in another thread. > > > > Finally, a word about projects that need testing with Neutron, advanced > > services and beyond. The same way the neutron-lib initiative is tackling > the > > python side of the Neutron codebase as of now, we'll have to work to > > identify the common pieces of functionality that will allow the reuse of > the > > testing machinery across multiple project, in a modular and decouple > > fashion, so that when Neutron related projects want to integrate with the > > Neutron CI engine, they will do so using good patterns rather than bad > ones; > > I believe that [3] is one of them, and that's why I have been pushing > back. > > can you explain what distinguishes good ones and bad ones? > > > > > Feedback welcome, as I am sure I may overlooked something. > > > > Cheers, > > Armando > > > > [1] https://wiki.openstack.org/wiki/Network/Testing > > [2] https://review.openstack.org/#/c/247697/ > > [3] https://review.openstack.org/#/c/242925/ > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
