This is really interesting discussion but was thrown off by the different use of ‘functional testing.’ I decided to reconcile it with my understanding and ended up with this two-pager. Sharing it in case it helps:
https://docs.google.com/document/d/1ILxfoJlov9lBfuuZtvwW_7bmlGaYYw0UsE1R9lo6XwQ/edit Regards, Mark Maglana QA/Release Engineer Nexus IS, Inc. Single Number Reach: 424-225-1309 On Mar 25, 2014, at 4:55 AM, Malini Kamalambal <malini.kamalam...@rackspace.com> wrote: > > > > We are talking about different levels of testing, > > 1. Unit tests - which everybody agrees should be in the individual project > itself > 2. System Tests - 'System' referring to (& limited to), all the components > that make up the project. These are also the functional tests for the > project. > 3. Integration Tests - This is to verify that the OS components interact > well and don't break other components -Keystone being the most obvious > example. This is where I see getting the maximum mileage out of Tempest. > > "Its not easy to detect what the integration points with other projects are, > any project can use any stable API from any other project. Because of this > all OpenStack APIs should fit into this category. " > > Any project can use any stable API –but that does not make all API tests , > Integration Tests. > A test becomes Integration test when it has two or more projects interacting > in the test. > > Individual projects should be held accountable to make sure that their API's > work – no matter who consumes them. > We should be able to treat the project as a complete system, make API calls > and validate that the response matches the API definitions. > Identifying issues earlier in the pipeline reduces the Total Cost of Quality. > > I agree that Integration Testing is hard. It is complicated because it > requires knowledge of how systems interact with each other – and knowledge > comes from a lot of time spent on analysis. > It requires people with project expertise to talk to each other & identify > possible test cases. > openstack-qa is the ideal forum to do this. > Holding projects responsible for their functionality will help the QA team > focus on complicated integration tests. > > "Having a second group writing tests to Nova's public APIs has been really > helpful in keeping us honest as well." > > Sounds like a testimonial for more project level testing :) > > > I see value in projects taking ownership of the System Tests - because if > the project is not 'functionally ready', it is not ready to integrate with > other components of Openstack. > > "What do you mean by not ready?" > > 'Functionally Ready' - The units that make up a project can work together as > a system, all API's have been exercised with positive & negative test cases > by treating the project as a complete system. > There are no known critical bugs. The point here being identify as many > issues as possible, earlier in the game. > > But for this approach to be successful, projects should have diversity in > the team composition - we need more testers who focus on creating these > tests. > This will keep the teams honest in their quality standards. > > As long as individual projects cannot guarantee functional test coverage, > we will need more tests in Tempest. > But that will shift focus away from Integration Testing, which can be done > ONLY in Tempest. > > Regardless of whatever we end up deciding, it will be good to have these > discussions sooner than later. > This will help at least the new projects to move in the right direction. > > -Malini > > > > > > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
