Thanks, Daryl. This is a complicated issue and I will try to spell out
my concern more clearly.
On 6/1/2012 6:19 PM, Daryl Walleck wrote:
Hi David,
The per test fixtures are there for two reasons. One - stability. If we were to
share one server among the tests, any of the previous tests tainted it or left
it in a bad state, the rest of the suite would fail for unclear reasons. The
second is for parallel execution. We cannot perform all those actions on a
single server at once.
These statements are obviously true but I am having trouble interpreting
them against your (below) expressed desire to have Tempest be an
acceptance test for a production deploy. If a test checked its results
in a way that made sure its server was not tainted or left in a bad
state, there would be no concern about running other tests on that
server. If a test does not do that, then it could "pass" but also leave
its server in a bad state. If we never reuse a server then such failures
will never be detected by Tempest.
I was really thinking about two things I noticed in the tests:
1. They spend a huge amount of time in server create, delete, resize,
rebuild, etc. even though those operations represent a miniscule
fraction of the code being tested.
2. There are a lot of tests that take the same path through the code but
vary parameters in a way that is not thorough enough to say that an API
is fully tested, and
that would be better covered by the unit tests that bypass the expensive
code that is the same for all the cases. The number of these cases
varies widely between
the different test classes in Tempest and I am not sure what principles
have been used to decide how many variants tempest should have.
I certainly agree that we should optimize Tempest as best we can. Tempest is an
black-box, integration test suite. In my view, its purpose is to throughly,
from end to end, test the integration of OpenStack components. For quick
testings, we have unit and component level integration tests. If those suites
are lacking, then more effort should be thrown behind those efforts as well.
I'm open to making all optimizations where we can, and I think there's still
quite a few things we can do. However, what I look for from Tempest is
confidence in making a production deploy of OpenStack, and out there are
corners I would rather not cut. I personally would not be comfortable
delegating tests for basic tasks such as resize, rebuild to run daily. What do
you think acceptable run times for a smoke grouping and full regression run
should be?
Daryl
For a full regression it should take as long as it needs to be run after
being optimized appropriately. I don't see a shortcut there. I was
imagining that a full regression run would happen every day. I don't
know enough about the inner workings of Jenkins to say how long a smoke
grouping should take but it is not just the time. If we managed to get
the time down by massive parallelization, it still might use an
unacceptable amount of (real) server resources.
I think it would be helpful to agree on:
a) How thorough tempest tests should be about variants of arguments for
both positive and negative tests in general, for full regression and smoke.
b) How (a) might be different for cases where the test cases take a long
time
c) Consider which, if any, cases it would make sense to use a pool of
pre-allocated servers rather than spinning up new ones for each test.
-David
On Jun 1, 2012, at 4:03 PM, David Kranz wrote:
I am a little confused about this. Most test classes define tearDownClass that
frees resources allocated in setUpClass. But two of the classes deviate from
this.
ServerActionsTest uses setUp and tearDown and creates a new server in setUp. I
think this means that a new server is created before running each test method.
This test is very slow, taking 9 minutes with three hefty compute nodes in a
cluster. Many of the methods could reuse the same server and the negative tests
don't need to create one at all. Unfortunately I think a lot of that time is
spent just doing resize. I think we should consider making this test be
nightly-build only.
ServerDetailsNegativeTest has methods that create lots of servers and has a
tearDown method that deletes them after each test method. That seems
unnecessary.
This test is very slow, taking 3 minutes on a cluster with three hefty compute
nodes. And that is with 15 tests being skipped pending bug fixes.
It also has a tearDownClass method that deletes all running servers whether
this test created them or not. That seems pretty bad. Why is it doing this?
Does any one have any comment about this?
-David
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to : openstack-qa-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack-qa-team
More help : https://help.launchpad.net/ListHelp
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to : openstack-qa-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack-qa-team
More help : https://help.launchpad.net/ListHelp
