About the patch I committed, in fact it's not moving. It's fixing. Because: 1. The logical reason is, currently in many cases(which I committed the patches) the account is already create per test case rather than per module. And if account is created in setUp() rather than setUpClass(), the clean up would be in tearDown() rather than tearDownClass(), because they're useless anyway after setUp create another account and more VMs. My patch didn't move account creation from setUpClass(). 2. The more direct reason is too many test cases in regression test failed just because lack of resource to create new deployment. E.g. https://issues.apache.org/jira/browse/CLOUDSTACK-3643 . Whenever I saw "Fail to create VPC" or "Fail to deploy VM", mostly it because the testing setup is overloaded.
I know VM creating is very time consuming, and reuse the account and VM is really nice! But the fact is many of current test cases is not written in this way. So I think we should release the resource as soon as it's obsolete. About reusing the already deployed VM and account, it would be a case-by-case issue. For example, TestAddVmToSubDomain case in test_accounts do create the two VMs shared across the module(class), then tear it down when in tearDownClass(). But sometime there is not a easy way to do so. I think that's why there are not many VirtualMachine.create() happened in setUpClass(). --Sheng On Tue, Jul 23, 2013 at 10:56 PM, Prasanna Santhanam <t...@apache.org> wrote: > In the test modules when you debug you will notice that accounts are > created once per module in setUpClass() all resources created within > it and tearDownClass() destroys the account initiating cleanup. All > the resources are appended to a cleanup [] list and deleted in > appropriate order at the end of the test module in tearDownClass() > > There are a few reasons for doing this for all the tests in the class > at once as opposed to doing it for every test. > > Modules are grouped by feature eg: test_tags does tags related > tests. And more often than not all tests in the module share similar > patterns in the set of API calls made to achieve a test scenario. > > If an account was created per test the overhead of cleaning up is much > higher than when cleaning up once per module. Because every new > account needs a new VR for the first VM deployed in it. And almost > every test will deploy a VM. So this slows down test run significantly > and eats up resources like VLANs very quickly which are needed for > every account. > > I saw a few fixes that moved the resource cleanup from tearDownClass() > to tearDown and that prompted me to send this email. Hope this makes > sense. I'd like to hear other's thoughts on how best to accomplish all > the tests in the most optimum way without hurting resources on a > deployment. > > Also - on the test infrastructure on jenkins.buildacloud.org there is > no way to timeout a specific test if it takes up more time than > necessary. So I'm going to introduce a timeout plugin in nose that > will abort the test if it takes longer than 1/2hr. I think this should > help weeding out test that do arbitrary 'sleeps' or wait for very long > cleanup operations. That way we should be able to optimize the test as > well > > Thoughts? > > -- > Prasanna., > > ------------------------ > Powered by BigRock.com > >
