> -----Original Message----- > From: Mice Xia [mailto:mice_...@tcloudcomputing.com] > Sent: Wednesday, January 30, 2013 9:23 PM > To: cloudstack-dev@incubator.apache.org > Subject: RE: [DISCUSS] Concerns on testing and improvements > > Prasanna: > > 2) More and more requests from contributors on how to write unit-tests > show up on our lists every week. > > I agree we need more unit tests. At this moment there isn’t much detailed > guideline or material about how to write a unit test with unified style. > For example, before 4.1.0, mocks are injected with MockComponentLocator > and these mock objects must be implemented manually. Later we bring > mockito back and simplify the test codes. Now with spring being introduced, > should we also use spring test framework?
Yes, that's why we need to merge javelin into master ASAP, writing unit test with Spring and Mockito and Junit/TestNG, is much easier. > And for different layers, the way to implement unit tests might be different, > to test a Resource class, we need a lot of mocks/stubs of external library; to > test a DAO, should we prepare the database with test data? > I think we need some consensus on ' what should be unit tested' and 'the > recommend way to write a unit test'. > > 4. Easing test writing is my primary concern at the moment and I've been > refactoring marvin to accomadate a easier dsl style test composition. I hope > to have a working test case sometime next week. > > I haven’t followed this for a while, and I don’t use devCloud . Are there any > documents describing how to run Marvin test with simulator/hypervisors? > > Regards > Mice > > -----Original Message----- > From: prasanna [mailto:srivatsav.prasa...@gmail.com] On Behalf Of > Prasanna Santhanam > Sent: Wednesday, January 30, 2013 2:02 AM > To: cloudstack-dev@incubator.apache.org > Subject: [DISCUSS] Concerns on testing and improvements > > On Tue, Jan 29, 2013 at 10:43:33AM -0500, Alex Huang wrote: > > > > It is also true CloudStack does not have a good automated > > > > regression test suite to make sure a check-in like this does not > > > > break some other features in CloudStack. But lack of a thorough > > > > automated regression suite a problem with CloudStack in general. > We've let in other big changes in this release. > > > > I know developers who wrote this check-in have thoroughly > > > > regression > > > tested > > > > to the best of their abilities. > > > > > > > > > > > This is a point I tried to make to a few folks before CloudStack > > > came here. If you want to provide a smooth path to committership you > > > need a solid regression testing framework (with near total coverage) > > > and an environment. Feeling blind and vulnerable to big changes like > > > this limits trust and we can't have a lack of trust. If you want > > > this project growing faster and healthier regression testing is key: it > > > has a > social impact. > > > > > > > +1 Again Agree on this. CloudStack has been lacking a good > > automated regression tests. We need to start doing something about > > it. > > This warrants its own thread (which unfortunately has become a common > feature on our lists) > > IMO - cloudstack is and remains largely untestable! I've often seen these > requests for more tests spring (no pun intended) up when a large code > merge is called for. Happened with api_refactoring as well. But the truth of > the matter is "tests are an afterthought". > > If you don't believe me: > > 1) One only needs look at the list of features waiting on ipclearance without > much to call in terms of tests. Feature is developed - testing can come later. > > 2) More and more requests from contributors on how to write unit-tests > show up on our lists every week. > > 3) Tests cannot magically appear unless there is a concerted effort in testing > right from feature conception which is lacking. > > There's another fundamental problem. Traditionally cloudstack development > has happened separate from testing processes. Code is 'thrown-over-the- > wall' to be tested which is not very different from how things are today. With > that attitude I only see test playing 'catch-up' to the almost *maddening* > pace of development. > > > And now for some solutions: > > 1. Start test development alongside feature development. Attach a test > engineer if possible to every feature branch. Make unit-test writing > mandatory for both committers and non-committers. > > 2. I've been working with some qa folk within my company to address the > issue more proactively. They (being non-committers) look forward to start > working on github [1] along with some of our engineers who wrote the > original integration tests. The test activity will proceed in the same feature > branch as being used by the developer. And will be connected to appropriate > infrastructure. But likely this will happen only after 4.1 > > 3. Cloudstack needs massive infrastructure, build and packaging pieces to > come together for doing live provider testing. While I got this up for 4.0 > with > a live environment, this hasn't received much love on master. Over the past > couple days I've been testing the replicability of this infrastructure and > have > been moving the pieces to builds.a.o > > 4. Easing test writing is my primary concern at the moment and I've been > refactoring marvin to accomadate a easier dsl style test composition. I hope > to have a working test case sometime next week. > > Much of this has come too little, too late. However I beseech others to pitch > in with ideas and contributions on improving testability on the whole to ease > this pain for the forthcoming releases. > > [1] https://github.com/cloudstack-qa > > -- > Prasanna., > PS: Forgot to break the thread earlier in haste.
