Sorry, pervious mail is off topic of BVT.
+1 to BVT
> -----Original Message-----
> From: Frank Zhang [mailto:frank.zh...@citrix.com]
> Sent: Tuesday, March 05, 2013 11:41 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: RE: [PROPOSAL] BVT for CloudStack checkins
>
> >> Many types of designs/applications are better tested by automating tests
> at the module/package level.
> >>In my experience, most coding errors are not because the code in a class
> was coded incorrectly but because the coder didn't understand how their
> class was supposed to work with other classes.
>
> If you guys remembered, I have argued with you intensely last month (in a
> lunch time, not in maillist), now I am happy to see you finally find out
> somebody supporting my idea.
> I would not hesitate to say again, the unit tests you guys thinking, talking,
> doing are INPRACTICAL and valueless.
> Testing on method level, mocking everything(current CloudStack doing, the
> idea you guys like) is commonly used in textbook and library project, it's not
> suitable for integration project like CloudStack. If you force people to do
> so,
> the only thing you get is tons of fake test cases.
>
> Again, I would say loudly that unit test in CloudStack should be in
> COMPONENT level. By Component level I mean, the test cases actual test
> workflow of a component, not to test if a 50 line method correctly returns a
> true or throws an exception.
> You can argue with me again that what I am saying is not unit test but
> integration test. Then I would say UNIT is not only a method or a class, the
> UNIT can be a component that is up to how your project defines it.
>
> Yesterday I heard from Min that our unit test doesn't allow to deployDB data
> because faking some DB data in test IS NOT unit test. The point sounds like a
> joke to me.
>
>
>
> > -----Original Message-----
> > From: Edison Su [mailto:edison...@citrix.com]
> > Sent: Tuesday, March 05, 2013 11:14 AM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: RE: [PROPOSAL] BVT for CloudStack checkins
> >
> > Recently, I read a topic about, "why automated test keeping
> > failing"(http://programmers.stackexchange.com/questions/185819/why-
> > does-automated-testing-keep-failing-in-my-company/185839#185839), the
> > comment is really resonated with me:
> >
> > "Most unit tests are questionable in value. Since the vast majority of
> > tests seem to be too simple."
> > "It is much harder to write good testable code than just working code."
> > "Many types of designs/applications are better tested by automating
> > tests at the module/package level. In my experience, most coding
> > errors are not because the code in a class was coded incorrectly but
> > because the coder didn't understand how their class was supposed to work
> with other classes."
> >
> > We'd better put our limited resource on BVT test on both simulator and
> > real hardware, at least, our BVT should cover all the hypervisors and
> > all the cloudstack APIs.
> >
> > So +1.
> >
> > > -----Original Message-----
> > > From: Alex Huang [mailto:alex.hu...@citrix.com]
> > > Sent: Tuesday, March 05, 2013 10:42 AM
> > > To: cloudstack-dev@incubator.apache.org
> > > Subject: [PROPOSAL] BVT for CloudStack checkins
> > >
> > > Hi All,
> > >
> > > As most of you are aware, the master branch keeps getting broken by
> > > checkins for various reasons. Committers need to be more
> > > responsible about their checkins but I don't think we can depend on that
> happening.
> > > There are various reasons. The most obvious to me is that granting
> > > committership is not based on code competency. (And I don't think
> > > it
> > > should.) Given that we need to build a BVT system for ensure that
> > > checkins do not break the branch.
> > >
> > > Here's my proposal:
> > >
> > > Existing components that we'll use.
> > >
> > > - Citrix has contributed its testing to Apache.
> > >
> > > - Apache CloudStack has already a simulator that's been used for
> scale
> > > testing.
> > >
> > > - Marvin
> > >
> > > - DevCloud-kvm
> > >
> > > Work Proposal:
> > >
> > > - Convert the Citrix testing into three phases:
> > >
> > > o Setup
> > >
> > > o Test
> > >
> > > o Verify
> > >
> > > - Add a Setup and Verify phase for the simulator
> > >
> > > - Add all of the agent commands necessary for the simulator to
> > > pass
> > the
> > > testing.
> > >
> > > - Add a Setup and Verify phase for devCloud-kvm
> > >
> > > - Add two more profiles to pom
> > >
> > > o Checkin-test-with-simulator: Runs the testing against the simulator
> > >
> > > o Checkin-test-with-devCloud: Runs the testing against devcloud
> > >
> > > - All of the profiles will attempt to also check the merge list
> > > that Chip
> > has
> > > proposed.'
> > >
> > > - We will also change marvin to easily add zones with actual
> > > hardware.
> > It
> > > will be based on a data driven document to do the setup.
> > >
> > > For a developer to checkin:
> > >
> > > - S/he must writes marvin tests for their feature and add it to
> > > the
> BVT.
> > >
> > > - S/he must run the checkin tests to verify everything works.
> > >
> > > - If the feature contains a hardware/vpx component, simulation
> must
> > be
> > > added.
> > >
> > > At this point, everything is about the developer writing in their
> > > feature branch and merging in.
> > >
> > > On infrastructure side:
> > >
> > > - We'll setup continuous BVT based on the simulator.
> > >
> > > - I again push that we must use Gerrit to test the code before
> > > it gets
> > > merge into the branch but I'll leave that for someone else to do that.
> > >
> > > Let me know what you guys think. I'll probably break out a bvt
> > > branch to work on this. Anyone want to join me?
> > >
> > > --Alex
