Recently, I read a topic about, "why automated test keeping failing"(http://programmers.stackexchange.com/questions/185819/why-does-automated-testing-keep-failing-in-my-company/185839#185839), the comment is really resonated with me:
"Most unit tests are questionable in value. Since the vast majority of tests seem to be too simple." "It is much harder to write good testable code than just working code." "Many types of designs/applications are better tested by automating tests at the module/package level. In my experience, most coding errors are not because the code in a class was coded incorrectly but because the coder didn't understand how their class was supposed to work with other classes." We'd better put our limited resource on BVT test on both simulator and real hardware, at least, our BVT should cover all the hypervisors and all the cloudstack APIs. So +1. > -----Original Message----- > From: Alex Huang [mailto:alex.hu...@citrix.com] > Sent: Tuesday, March 05, 2013 10:42 AM > To: cloudstack-dev@incubator.apache.org > Subject: [PROPOSAL] BVT for CloudStack checkins > > Hi All, > > As most of you are aware, the master branch keeps getting broken by > checkins for various reasons. Committers need to be more responsible > about their checkins but I don't think we can depend on that happening. > There are various reasons. The most obvious to me is that granting > committership is not based on code competency. (And I don't think it > should.) Given that we need to build a BVT system for ensure that checkins > do not break the branch. > > Here's my proposal: > > Existing components that we'll use. > > - Citrix has contributed its testing to Apache. > > - Apache CloudStack has already a simulator that's been used for > scale > testing. > > - Marvin > > - DevCloud-kvm > > Work Proposal: > > - Convert the Citrix testing into three phases: > > o Setup > > o Test > > o Verify > > - Add a Setup and Verify phase for the simulator > > - Add all of the agent commands necessary for the simulator to pass > the > testing. > > - Add a Setup and Verify phase for devCloud-kvm > > - Add two more profiles to pom > > o Checkin-test-with-simulator: Runs the testing against the simulator > > o Checkin-test-with-devCloud: Runs the testing against devcloud > > - All of the profiles will attempt to also check the merge list that > Chip has > proposed.' > > - We will also change marvin to easily add zones with actual > hardware. It > will be based on a data driven document to do the setup. > > For a developer to checkin: > > - S/he must writes marvin tests for their feature and add it to the > BVT. > > - S/he must run the checkin tests to verify everything works. > > - If the feature contains a hardware/vpx component, simulation must > be > added. > > At this point, everything is about the developer writing in their feature > branch and merging in. > > On infrastructure side: > > - We'll setup continuous BVT based on the simulator. > > - I again push that we must use Gerrit to test the code before it > gets > merge into the branch but I'll leave that for someone else to do that. > > Let me know what you guys think. I'll probably break out a bvt branch to > work on this. Anyone want to join me? > > --Alex
