On 5/30/2012 3:56 PM, Jay Pipes wrote:
> With ubuntu and several other full Linux OSes, the best I've seen
is 15 minute smoke tests and 40-ish minute full regression runs, which
in my case is within my pain threshold. In terms of actions, we've paid
the majority of the time price for testing scenarios. The additional
tests I'm adding (such as the SSH code) don't add significant time to
tests, but keep adding value. The same can be said of checking for
events that should be generated on server actions, and other side
effects that take minimal time to run. As we start to move in that
direction, test classes will become more streamlined/optimized. The test
class I added with the initial SSH tests is a good example of this, and
I'd be more than glad to share some other examples of how we can do more
with less. Eventually these types of optimizations can grow into our
coding standards.
What's everyone else's feelings on smoke test pain point times? I
think
getting a sense of that will make any further decisions a bit more
clear.
Regardless of what length of time we say, I still think we need to get
parallel execution working properly (with the existing --processes
option in nosetest's multiprocessing plugin).
Agreed. I think like we talked about at the conference, I think this
discussion and the overall discussion on optimization would probably be
better handled in a Skype meeting where we can all discuss this in real
time. I'm free every day this week save Friday.
Cool, I will be at the meeting tomorrow, but otherwise not really
available much this week (have my folks in town). Next week, though,
I'm all yours. :) Let's set up a time to Skype when we're at the IRC
meeting tomorrow.
Best,
-jay
Though I agree with the sentiment here, I think we also need to pay more
attention to the relationship between tempest and unit tests in the
projects. There are currently >60 calls to create_server in the tempest
tests. I didn't check to see if any were inside loops. This is just
going to be slow, and get worse. Perhaps we should try to avoid creating
servers when possible. The main difference between the nova unit tests
and tempest seems to be that tempest really uses the hypervisor and is
more anal about combinations and return codes and api details. I think
there are some tempest tests with multiple variants of parameters but
where the difference in code path is already covered by the unit tests,
or should be.
Perhaps tempest should only have one case for such families of calls but
make sure that the unit tests are an adequate substitute.
-David
--
Mailing list: https://launchpad.net/~openstack-qa-team
Post to : openstack-qa-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack-qa-team
More help : https://help.launchpad.net/ListHelp
