We do that, but the majority of what's useful to test is advanced zone stuff. Which we can also do in devcloud with the right config. That's sort of what I was getting at, that we should standardize on an advanced config for devcloud so that everyone is writing against and testing the same.
On Thu, Jan 31, 2013 at 12:29 AM, Ahmad Emneina <aemne...@gmail.com> wrote: > Can we not group the tests vs standardizing all test to work against > devcloud? that way one can run the devcloud test suite against devcloud and > not be bothered with failing advanced zone tests. > > > On Wed, Jan 30, 2013 at 11:03 PM, Marcus Sorensen <shadow...@gmail.com>wrote: > >> On the integration testing, my impressions are that they're written >> against a specific environment. That's probably expected, being an >> actual functionality test, but it makes it hard to actually run them. >> Some of them work against devcloud in a basic zone, others work for an >> advanced setup, some tests don't seem to work at all and I'm not sure >> if they're just unfinished or specific to someone's test environment. >> Everyone might be writing tests but they only work in their >> environment. >> >> I've mentioned this before, but it should be possible to standardize >> on a marvin config that is self-contained (management server inside), >> that anyone can download and run the tests on, write tests against. >> Even possibly have a devcloud vm that pulls down master on a cron job, >> runs the tests, and emails/reports the results every day. It should >> even be possible to have a devcloud and devcloud-kvm with the same >> configs (same address ranges, etc) so the tests can be run against >> both. >> >> On Wed, Jan 30, 2013 at 11:25 PM, Edison Su <edison...@citrix.com> wrote: >> > >> > >> >> -----Original Message----- >> >> From: Mice Xia [mailto:mice_...@tcloudcomputing.com] >> >> Sent: Wednesday, January 30, 2013 9:23 PM >> >> To: cloudstack-dev@incubator.apache.org >> >> Subject: RE: [DISCUSS] Concerns on testing and improvements >> >> >> >> Prasanna: >> >> >> >> 2) More and more requests from contributors on how to write unit-tests >> >> show up on our lists every week. >> >> >> >> I agree we need more unit tests. At this moment there isn’t much >> detailed >> >> guideline or material about how to write a unit test with unified style. >> >> For example, before 4.1.0, mocks are injected with MockComponentLocator >> >> and these mock objects must be implemented manually. Later we bring >> >> mockito back and simplify the test codes. Now with spring being >> introduced, >> >> should we also use spring test framework? >> > >> > >> > Yes, that's why we need to merge javelin into master ASAP, writing unit >> test with Spring and Mockito and Junit/TestNG, is much easier. >> > >> >> And for different layers, the way to implement unit tests might be >> different, >> >> to test a Resource class, we need a lot of mocks/stubs of external >> library; to >> >> test a DAO, should we prepare the database with test data? >> > >> >> I think we need some consensus on ' what should be unit tested' and 'the >> >> recommend way to write a unit test'. >> >> >> >> 4. Easing test writing is my primary concern at the moment and I've been >> >> refactoring marvin to accomadate a easier dsl style test composition. I >> hope >> >> to have a working test case sometime next week. >> >> >> >> I haven’t followed this for a while, and I don’t use devCloud . Are >> there any >> >> documents describing how to run Marvin test with simulator/hypervisors? >> >> >> >> Regards >> >> Mice >> >> >> >> -----Original Message----- >> >> From: prasanna [mailto:srivatsav.prasa...@gmail.com] On Behalf Of >> >> Prasanna Santhanam >> >> Sent: Wednesday, January 30, 2013 2:02 AM >> >> To: cloudstack-dev@incubator.apache.org >> >> Subject: [DISCUSS] Concerns on testing and improvements >> >> >> >> On Tue, Jan 29, 2013 at 10:43:33AM -0500, Alex Huang wrote: >> >> > > > It is also true CloudStack does not have a good automated >> >> > > > regression test suite to make sure a check-in like this does not >> >> > > > break some other features in CloudStack. But lack of a thorough >> >> > > > automated regression suite a problem with CloudStack in general. >> >> We've let in other big changes in this release. >> >> > > > I know developers who wrote this check-in have thoroughly >> >> > > > regression >> >> > > tested >> >> > > > to the best of their abilities. >> >> > > > >> >> > > > >> >> > > This is a point I tried to make to a few folks before CloudStack >> >> > > came here. If you want to provide a smooth path to committership you >> >> > > need a solid regression testing framework (with near total coverage) >> >> > > and an environment. Feeling blind and vulnerable to big changes like >> >> > > this limits trust and we can't have a lack of trust. If you want >> >> > > this project growing faster and healthier regression testing is >> key: it has a >> >> social impact. >> >> > > >> >> > >> >> > +1 Again Agree on this. CloudStack has been lacking a good >> >> > automated regression tests. We need to start doing something about >> >> > it. >> >> >> >> This warrants its own thread (which unfortunately has become a common >> >> feature on our lists) >> >> >> >> IMO - cloudstack is and remains largely untestable! I've often seen >> these >> >> requests for more tests spring (no pun intended) up when a large code >> >> merge is called for. Happened with api_refactoring as well. But the >> truth of >> >> the matter is "tests are an afterthought". >> >> >> >> If you don't believe me: >> >> >> >> 1) One only needs look at the list of features waiting on ipclearance >> without >> >> much to call in terms of tests. Feature is developed - testing can come >> later. >> >> >> >> 2) More and more requests from contributors on how to write unit-tests >> >> show up on our lists every week. >> >> >> >> 3) Tests cannot magically appear unless there is a concerted effort in >> testing >> >> right from feature conception which is lacking. >> >> >> >> There's another fundamental problem. Traditionally cloudstack >> development >> >> has happened separate from testing processes. Code is 'thrown-over-the- >> >> wall' to be tested which is not very different from how things are >> today. With >> >> that attitude I only see test playing 'catch-up' to the almost >> *maddening* >> >> pace of development. >> >> >> >> >> >> And now for some solutions: >> >> >> >> 1. Start test development alongside feature development. Attach a test >> >> engineer if possible to every feature branch. Make unit-test writing >> >> mandatory for both committers and non-committers. >> >> >> >> 2. I've been working with some qa folk within my company to address the >> >> issue more proactively. They (being non-committers) look forward to >> start >> >> working on github [1] along with some of our engineers who wrote the >> >> original integration tests. The test activity will proceed in the same >> feature >> >> branch as being used by the developer. And will be connected to >> appropriate >> >> infrastructure. But likely this will happen only after 4.1 >> >> >> >> 3. Cloudstack needs massive infrastructure, build and packaging pieces >> to >> >> come together for doing live provider testing. While I got this up for >> 4.0 with >> >> a live environment, this hasn't received much love on master. Over the >> past >> >> couple days I've been testing the replicability of this infrastructure >> and have >> >> been moving the pieces to builds.a.o >> >> >> >> 4. Easing test writing is my primary concern at the moment and I've been >> >> refactoring marvin to accomadate a easier dsl style test composition. I >> hope >> >> to have a working test case sometime next week. >> >> >> >> Much of this has come too little, too late. However I beseech others to >> pitch >> >> in with ideas and contributions on improving testability on the whole >> to ease >> >> this pain for the forthcoming releases. >> >> >> >> [1] https://github.com/cloudstack-qa >> >> >> >> -- >> >> Prasanna., >> >> PS: Forgot to break the thread earlier in haste. >>
