I disagree. Error handling should be part of our testing. We should incorporate the simulator into the BVT and regression tests. On testcases that really is to test the business logic rather than the provisioning code, the test case should perform all of the provisioning on the simulator instead. Then simulator can be programmed to simulate VM stopped failure etc and see how the business responds to these problems.
--Alex > -----Original Message----- > From: Anthony Xu [mailto:xuefei...@citrix.com] > Sent: Thursday, July 18, 2013 3:02 PM > To: dev@cloudstack.apache.org > Subject: RE: [rant] stupid test cases > > +1 VM can be in "Stopped" state > > > Anthony > > -----Original Message----- > From: Marcus Sorensen [mailto:shadow...@gmail.com] > Sent: Wednesday, July 17, 2013 10:47 PM > To: dev@cloudstack.apache.org > Subject: Re: [rant] stupid test cases > > I can understand that we may want to test that everything related to the > domain gets cleaned up properly. We have run into all sorts of things when > deleting accounts, for example where resources won't clean up because the > account is gone and we throw null pointers because a bunch of code looks up > account when deleting. However, to your point, VMs can be created in a > "stopped" state and that wouldn't incur the overhead of deployment. > On Jul 17, 2013 11:33 PM, "Prasanna Santhanam" <t...@apache.org> wrote: > > > I was just going through one of the automated test cases and I find it > > really silly that there's the following test: > > > > def test_forceDeleteDomain(self): > > """ Test delete domain force option""" > > > > # Steps for validations > > # 1. create a domain DOM > > # 2. create 2 users under this domain > > # 3. deploy 1 VM into each of these user accounts > > # 4. create PF / FW rules for port 22 on these VMs for their > > # respective accounts > > # 5. delete the domain with force=true option > > # Validate the following > > # 1. listDomains should list the created domain > > # 2. listAccounts should list the created accounts > > # 3. listvirtualmachines should show the Running VMs > > # 4. PF and FW rules should be shown in listFirewallRules > > # 5. domain should delete successfully and above three list calls > > # should show all the resources now deleted. listRouters should > > # not return any routers in the deleted accounts/domains > > > > Why would one need the overhead of creating VMs in a domain deletion > test? > > Do > > we not understand that the basics accounts/domains/ etc that > > cloudstack has nothing to do with the virtual machines? This kind of a > > test slows down other useful tests that we could be running. More over > > when this fails in the VM creation step I'd have to go in and analyse > > logs to realize that deletedomain was perhaps fine but vm creation has > > failed. > > > > That's a pointless effort. I'm sure there are others in the automated > > tests that do this kind of wasteful testing. So please please pleaes > > &*()#@()# please review test plans before automating them! > > > > I'm not going to be looking at this forever to fix these issues when > > we want to see pretty metrics and numbers. > > > > -- > > Prasanna., > > > > ------------------------ > > Powered by BigRock.com > > > >
