On 03/30/2015 10:44 AM, Matthew Treinish wrote:
On Mon, Mar 30, 2015 at 12:21:18PM +0530, Rohan Kanade wrote:
Since tests can now be removed from Tempest <
https://wiki.openstack.org/wiki/QA/Tempest-test-removal> and migrated to
their specific projects.
Does Tempest plan to discover/run these tests in tempest gates? If yes, how
is that going to be done? Will there be a discovery mechanism in Tempest
to discover tests from individual projects?
No, the idea behind that wiki page is to outline the procedure for finding
something that is out of scope and doesn't belong in tempest and is also safe
to remove from the tempest jobs. The point of going through that entire
procedure is that the test being removed should not be run in the tempest gates
anymore and will become the domain of the other project.
Also, IMO the moved test ideally won't be in the same pattern of a tempest test
or have the same constraints of a tempest test and would ideally be more coupled
to the project under test's internals. So that wouldn't be appropriate to
include in a tempest run either.
For example, the first test we removed with that procedure was:
https://review.openstack.org/#/c/158852/
which removed the flavor negative tests from tempest. These were just testing
operations that would go no deeper than Nova's DB layer. Which was something
we couldn't verify in tempest. They also didn't really belong in tempest because
they were just implicitly verifying Nova's DB layer through API responses. The
replacement tests:
http://git.openstack.org/cgit/openstack/nova/tree/nova/tests/functional/wsgi/test_flavor_manage.py
were able to verify the state of the DB was correct and ensure the correct
behavior both in the api and nova's internals. This kind of testing is something
which doesn't belong in tempest or any other external test suite. It is also
what I feel we should be targeting for with project specific in-tree functional
testing and the kind of thing we should be using the removal process on that
wiki page for.
-Matt Treinish
Matt, while everything you say here is true, I don't think it answers
the whole question. neutron is also planning to move the tempest
networking tests into the neutron repo with safeguards to prevent
incompatible changes, but also keeping the tests in a form that is not
so different from tempest.
The problem is that deployers/users/refstack/etc. (let's call them
verifiers) want an "OpenStack functional verification suite". Until now
that has been easy since most of what that requires is in tempest, and
Rally calls tempest. But to a verifier, the fact that all the tests used
for verification are in one tempest repo is an implementation detail.
OpenStack verifiers do not want to lose neutron tests because they moved
out of tempest. So verifiers will need to do something about this and it
would be better if we all did it as a community by agreeing on a UX and
method for locating and running all the tests that should be included in
an OpenStack functional test suite. Even now, there are tests that are
useful for verification that are not in tempest.
I think the answer that Boris gave
http://lists.openstack.org/pipermail/openstack-dev/2015-March/060173.html is
trying to address this by saying that Rally will take on the role of
being the "OpenStack verification suite" (including performance tests).
I don't know if that is the best answer and tempest/rally could agree on
a UX/discovery/import mechanism, but I think we are looking at one of
those two choices.
-David
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
