Even as a core contributor for several years, it has never been clear
what the scope of these tests should be.
As we move forward with the necessity of moving functional testing to
projects, we need to answer this question for real, understanding that
part of the mission for these tests now is validation of clouds. Doing
so is made difficult by the fact that the tempest api tests take a very
opinionated view of how services are invoked. In particular, the tempest
client is very low-level and at present the way a functional test is
written depends on how and where it is going to run.
In an ideal world, functional tests could execute in a variety of
environments ranging from those that completely bypass wsgi layers and
make project api calls directly, to running in a fully integrated "real"
environment as the tempest tests currently do. The challenge is that
there are mismatches between how the tempest client looks to test code
and how doing object-model api calls looks. Most of this discrepancy is
because many pieces of invoking a service are hard-coded into the tests
rather than being abstracted in a client. Some examples are:
1. Response validation
2. json serialization/deserialization
3. environment description (tempest.conf)
4. Forced usage of addCleanUp
Maru Newby and I have proposed changing the test code to use a more
abstract client by defining the expected signature and functionality
of methods on the client. Roughly, the methods would take positional
arguments for pieces that go in the url part of a REST call, and kwargs
for the json payload. The client would take care of these enumerated
issues (if necessary) and return an attribute dict. The test code itself
would then just be service calls and checks of returned data. Returned
data would be inspected as resource.id instead of resource['id']. There
is a strawman example of this for a few neutron apis here:
https://review.openstack.org/#/c/106916/
Doing this would have the twin advantages of eliminating the need for
boilerplate code in tests and making it possible to run the tests in
different environments. It would also allow the inclusion of project
functional tests in more general validation scenarios.
Since we are proposing to move parts of tempest into a stable library
https://review.openstack.org/108858, we need to define the client in a
way that meets all the needs outlined here before doing so. The actual
work of defining the client in tempest and changing the code that uses
it could largely be done one service at a time, in the tempest code,
before being split out.
What do folks think about this idea?
-David
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
