Would like provide some perspective on testing efforts done. All test plans, results, automation has been posted to community and I took the data from that only[1].
Thanks for raising the issues around testing before code check-in. This would help in all fronts esp frontloading quality. Enabling development to run tests require good test suites and community has been helping with patches to fix failing test cases and adding new test suites. I hope this would be taken seriously and feature owners check in test code religiously. -New test plans developed and posted on cwiki Around 5500+ test cases ran manually [1] Another 2000+ test cases ran through automation across all HV, with multiple configurations [2] - Logged defects and verified defects. Created 2200 , resolved 1850, verified 1650 Defects. - several have helped to run automation 6 configurations for BVTs with Xen, KVM and VMWare on various configurations and 6 more additional configurations are being run for regression covering HVs and various configurations [1] https://cwiki.apache.org/confluence/display/CLOUDSTACK/QA+-+4.2+Test+Execution+Results [2]https://cwiki.apache.org/confluence/display/CLOUDSTACK/ACS+4.2+Automation+Results+-+Baseline Hope these discussions would help to put focus on quality and bring in contributors for automation efforts. There are several features which people can just pick up and run with it. Even today there are several automation patches pending on review board. If people can pick these up and help to review , fix and submit it would help everyone. -----Original Message----- From: Daan Hoogland [mailto:daan.hoogl...@gmail.com] Sent: Monday, September 09, 2013 1:48 PM To: dev Subject: Re: [VOTE] Apache CloudStack 4.2.0 (fourth round) Why can't we cover every use case, Marcus. We will need help from users, but if they do help it will be easy to do so. On Mon, Sep 9, 2013 at 10:43 PM, Marcus Sorensen <shadow...@gmail.com> wrote: > I was actually talking about separate things in relation to this > thread and the other where I mentioned that I'd like to see a release > focused on bugfixing and testing. With that, I'm advocating a test for > every api call and focusing on broadening use case test coverage. > > Here, I'm simply talking about taking the support matrix and doing > some vary basic testing. This can be a dozen or so tests, each > platform we say we support needs to successfully deploy a zone and a > vm on every storage type that is in the support matrix. I don't think > this would include plugins (or maybe those are left to the developer > of the third party plugin). For KVM, this is literally a marvin script > away from being there, I don't think there's a ton of work. I have no > idea what we have or can do with vmware, and I'm guessing Xen is > largely covered already. > > We'll never be able to cover every use case, I may be able to deploy a > zone with my KVM setup, but not someone else's special network layout. > I'd just like to see sanity checks to say it works, at all, on the > handful of 'supported' systems. > > On Mon, Sep 9, 2013 at 2:24 PM, Mike Tutkowski > <mike.tutkow...@solidfire.com> wrote: >> Do we have any statistics that say how many of our customers are >> using feature x, feature y, etc.? >> >> If not, I would say if we know about a feature that has regressed to >> the point of breakage in 4.2 that it should be fixed before releasing >> (or at the very least well documented, so - if it is impactful to >> someone - they do not upgrade until it has been fixed). >> >> >> On Mon, Sep 9, 2013 at 2:12 PM, Chiradeep Vittal < >> chiradeep.vit...@citrix.com> wrote: >> >>> I think that Animesh is trying to stress what is "key". If it hits >>> 1% of cloud operators is it key? >>> >>> >>> On 9/9/13 7:42 AM, "Simon Weller" <swel...@ena.com> wrote: >>> >>> >-1 from me as well. >>> > >>> > >>> >I know we're trying to hit timed releases, but I think it's very >>> >important to preserve key underlying functionality across releases. >>> >If a supported and documented feature is known to be broken, we >>> >need to address it...if we don't, it's going to cause lots of pain, >>> >and reflect badly on ACS as a project. >>> > >>> >----- Original Message ----- >>> > >>> >From: "Chip Childers" <chip.child...@sungard.com> >>> >To: dev@cloudstack.apache.org >>> >Sent: Monday, September 9, 2013 9:24:23 AM >>> >Subject: Re: [VOTE] Apache CloudStack 4.2.0 (fourth round) >>> > >>> >On Sun, Sep 08, 2013 at 12:40:30AM -0600, Marcus Sorensen wrote: >>> >> -1 ... sorry guys, especially with Simon chiming in. >>> >> >>> >> I'd request f2c5b5fbfe45196dfad2821fca513ddd6efa25c9 be cherry-picked. >>> > >>> >Agreed. >>> > >>> >I'm -1, given simon's perspective as well. Since we have the fix, >>> >let's get it into the release. >>> > >>> >>> >> >> >> -- >> *Mike Tutkowski* >> *Senior CloudStack Developer, SolidFire Inc.* >> e: mike.tutkow...@solidfire.com >> o: 303.746.7302 >> Advancing the way the world uses the >> cloud<http://solidfire.com/solution/overview/?video=play> >> *(tm)*
