On Aug 11, 2014, at 12:00 PM, David Kranz <dkr...@redhat.com> wrote:
> On 08/06/2014 05:48 PM, John Griffith wrote: >> I have to agree with Duncan here. I also don't know if I fully understand >> the limit in options. Stress test seems like it could/should be different >> (again overlap isn't a horrible thing) and I don't see it as siphoning off >> resources so not sure of the issue. We've become quite wrapped up in >> projects, programs and the like lately and it seems to hinder forward >> progress more than anything else. >> >> I'm also not convinced that Tempest is where all things belong, in fact I've >> been thinking more and more that a good bit of what Tempest does today >> should fall more on the responsibility of the projects themselves. For >> example functional testing of features etc, ideally I'd love to have more of >> that fall on the projects and their respective teams. That might even be >> something as simple to start as saying "if you contribute a new feature, you >> have to also provide a link to a contribution to the Tempest test-suite that >> checks it". Sort of like we do for unit tests, cross-project tracking is >> difficult of course, but it's a start. The other idea is maybe functional >> test harnesses live in their respective projects. >> >> Honestly I think who better to write tests for a project than the folks >> building and contributing to the project. At some point IMO the QA team >> isn't going to scale. I wonder if maybe we should be thinking about >> proposals for delineating responsibility and goals in terms of functional >> testing? >> >> >> > All good points. Your last paragraph was discussed by the QA team leading up > to and at the Atlanta summit. The conclusion was that the api/functional > tests focused on a single project should be part of that project. As Sean > said, we can envision there being half (or some other much smaller number) as > many such tests in tempest going forward. > > Details are under discussion, but the way this is likely to play out is that > individual projects will start by creating their own functional tests outside > of tempest. Swift already does this and neutron seems to be moving in that > direction. There is a spec to break out parts of tempest > (https://github.com/openstack/qa-specs/blob/master/specs/tempest-library.rst) > into a library that might be used by projects implementing functional tests. > > Once a project has "sufficient" functional testing, we can consider removing > its api tests from tempest. This is a bit tricky because tempest needs to > cover *all* cross-project interactions. In this respect, there is no clear > line in tempest between scenario tests which have this goal explicitly, and > api tests which may also involve interactions that might not be covered in a > scenario. So we will need a principled way to make sure there is complete > cross-project coverage in tempest with a smaller number of api tests. > > -David We need to be careful about dumping the tests from tempest now that the DefCore group is relying on them as well. Tempest is no longer just a developer/QA/operations tool. It’s also being used as the basis of a trademark enforcement tool. That’s not to say we can’t change the test suite, but we have to consider a new angle when doing so. Doug > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
