>> Many types of designs/applications are better tested by automating tests at >> the module/package level. >>In my experience, most coding errors are not because the code in a class was >>coded incorrectly but because the coder didn't understand how their class was >>supposed to work with other classes.
If you guys remembered, I have argued with you intensely last month (in a lunch time, not in maillist), now I am happy to see you finally find out somebody supporting my idea. I would not hesitate to say again, the unit tests you guys thinking, talking, doing are INPRACTICAL and valueless. Testing on method level, mocking everything(current CloudStack doing, the idea you guys like) is commonly used in textbook and library project, it's not suitable for integration project like CloudStack. If you force people to do so, the only thing you get is tons of fake test cases. Again, I would say loudly that unit test in CloudStack should be in COMPONENT level. By Component level I mean, the test cases actual test workflow of a component, not to test if a 50 line method correctly returns a true or throws an exception. You can argue with me again that what I am saying is not unit test but integration test. Then I would say UNIT is not only a method or a class, the UNIT can be a component that is up to how your project defines it. Yesterday I heard from Min that our unit test doesn't allow to deployDB data because faking some DB data in test IS NOT unit test. The point sounds like a joke to me. > -----Original Message----- > From: Edison Su [mailto:edison...@citrix.com] > Sent: Tuesday, March 05, 2013 11:14 AM > To: cloudstack-dev@incubator.apache.org > Subject: RE: [PROPOSAL] BVT for CloudStack checkins > > Recently, I read a topic about, "why automated test keeping > failing"(http://programmers.stackexchange.com/questions/185819/why- > does-automated-testing-keep-failing-in-my-company/185839#185839), the > comment is really resonated with me: > > "Most unit tests are questionable in value. Since the vast majority of tests > seem to be too simple." > "It is much harder to write good testable code than just working code." > "Many types of designs/applications are better tested by automating tests at > the module/package level. In my experience, most coding errors are not > because the code in a class was coded incorrectly but because the coder > didn't understand how their class was supposed to work with other classes." > > We'd better put our limited resource on BVT test on both simulator and real > hardware, at least, our BVT should cover all the hypervisors and all the > cloudstack APIs. > > So +1. > > > -----Original Message----- > > From: Alex Huang [mailto:alex.hu...@citrix.com] > > Sent: Tuesday, March 05, 2013 10:42 AM > > To: cloudstack-dev@incubator.apache.org > > Subject: [PROPOSAL] BVT for CloudStack checkins > > > > Hi All, > > > > As most of you are aware, the master branch keeps getting broken by > > checkins for various reasons. Committers need to be more responsible > > about their checkins but I don't think we can depend on that happening. > > There are various reasons. The most obvious to me is that granting > > committership is not based on code competency. (And I don't think it > > should.) Given that we need to build a BVT system for ensure that > > checkins do not break the branch. > > > > Here's my proposal: > > > > Existing components that we'll use. > > > > - Citrix has contributed its testing to Apache. > > > > - Apache CloudStack has already a simulator that's been used for > > scale > > testing. > > > > - Marvin > > > > - DevCloud-kvm > > > > Work Proposal: > > > > - Convert the Citrix testing into three phases: > > > > o Setup > > > > o Test > > > > o Verify > > > > - Add a Setup and Verify phase for the simulator > > > > - Add all of the agent commands necessary for the simulator to pass > the > > testing. > > > > - Add a Setup and Verify phase for devCloud-kvm > > > > - Add two more profiles to pom > > > > o Checkin-test-with-simulator: Runs the testing against the simulator > > > > o Checkin-test-with-devCloud: Runs the testing against devcloud > > > > - All of the profiles will attempt to also check the merge list > > that Chip > has > > proposed.' > > > > - We will also change marvin to easily add zones with actual > > hardware. > It > > will be based on a data driven document to do the setup. > > > > For a developer to checkin: > > > > - S/he must writes marvin tests for their feature and add it to > > the BVT. > > > > - S/he must run the checkin tests to verify everything works. > > > > - If the feature contains a hardware/vpx component, simulation must > be > > added. > > > > At this point, everything is about the developer writing in their > > feature branch and merging in. > > > > On infrastructure side: > > > > - We'll setup continuous BVT based on the simulator. > > > > - I again push that we must use Gerrit to test the code before it > > gets > > merge into the branch but I'll leave that for someone else to do that. > > > > Let me know what you guys think. I'll probably break out a bvt branch > > to work on this. Anyone want to join me? > > > > --Alex
