I agree that we should be relying on automated tests. It's too hit or miss (not to mention slow) when we put so much weight on semi random testing.
On Wed, Apr 29, 2015 at 8:12 PM, David Nalley <da...@gnsa.us> wrote: > The problem is really two fold. > Cutting an RC isn't terribly time consuming, once you've done it a > dozen times or so, it goes by pretty quickly. > > The problem is the time it takes to do any material testing; and that > we don't have a way of telling what the status of a candidate is > without doing lots of manual testing. We've had multiple cases of > having enough binding votes for 4.5.x candidates only for someone to > find problems. This leads me (and this could just be my perspective) > to let an RC sit for a while - get folks to test it. My experience as > a person voting is that rapid churn of RCs leads me to not even > attempt to setup a testing environment, much less do any testing. I > found that by the time I could get around to testing an RC vote had > been cancelled and it had been rerolled, sometimes twice. > > I've said this before, and I'll toss it out again. I think we have to > start 'trusting' the automated tests. We find lots of problems > manually, but we don't improve the testing situation for those blocker > issues that stopped us from releasing. I'm of the opinion we should > release quickly, very quickly. But if we have to spend a week just > looking for bugs, that will never fly. So - let the standard be > automated tests - and if there's a problem, we create an automated > test for it, and solve in a week or two. That will force us to live > and die by automated tests. > > --David > > On Wed, Apr 29, 2015 at 9:45 AM, Rohit Yadav <rohit.ya...@shapeblue.com> > wrote: > > Hi Wilder - Thanks we’ll need all the ammo we have :) I’ve already done > my testings wrt XS 6.2/6.5 and KVM qemu 2.0/2.1/2.2 so will test KVM 1.5 > and VMware 5.5. > > > > Hi Daan - makes sense, we’re “so" late with 4.5.1 that we need to focus > on 4.6.0 to avoid delaying it. I’ve asked David regarding co-piloting the > next 4.5.1 RC since he could be busy and I want to help with lifting some > weights. I’m planning to do basic tests with VMware today, publish new > systemvmtemplates tomorrow (I’m already done with KVM and Xen) and plan to > cut a 4.5.1 RC on Monday and if we don’t hit any blocker or regressions ACS > 4.5.1 should be released by the end of next week. > > > >> On 29-Apr-2015, at 2:49 pm, Wilder Rodrigues < > wrodrig...@schubergphilis.com> wrote: > >> > >> Hi Rohit, > >> > >> I will join you in testing 4.5 form next week - just have to finish > some stuff. > >> > >> My environments will be: > >> > >> * Xen 6.2/6.5 > >> * KVM (qemu 1.5.3 and 2.1) > >> > >> Let’s rock! > >> > >> Cheers, > >> Wilder > >> > >> On 28 Apr 2015, at 18:33, Rohit Yadav <rohit.ya...@shapeblue.com> > wrote: > >> > >>> Hi all, > >>> > >>> ### 4.5 Release Effort > >>> > >>> I’ve been exhaustively testing ACS 4.5 wrt Xen 6.2, Xen 6.5, KVM (qemu > 2.0, 2.3) and I feel we’re pretty good but we need testing efforts and I > want to help drive efforts to releasing ACS 4.5.1 -- if you’ve any issues > that you would like to get fixed that are either regressions or blockers > can you please share on this thread? If we don’t find any blockers or > regressions let us invest in ACS 4.5 testing and release it soon. For other > minor issues we can always fix them and release 4.5.2 etc if and when > needed in future. > >>> > >>> Regarding my 4.5 testing - I’ve tested basic vm life cycle operations > (deploy, start, stop, destroy/expunge, migrate to another host, migrate to > another storage pool, deploy using iso) for all cases and found no issues; > >>> > >>> KVM with basic zone (with/without security groups, SG blockers were > found and fixed) > >>> KVM with Adv zone (with/without security groups, both isolated network > and VPC worked, networking issues were found and fixed) > >>> XenServer 6.2 with basic and adv zones (with/without security groups, > no SG issues found) > >>> XenServer 6.5 with basic and adv zones (with/without security groups, > SG blockers found and fixed) > >>> > >>> I found one issue that could be a blocker (if it’s not a hardware/env > issue) -- when using security groups in advance zone with a dedicated vlan > id (say vlan://500), I was unable to access the VMs (CPVM, SSVM, VR or user > vms) though all normal vm_life cycles seems to work. Though this worked for > me when I used the vlan://untagged. I tried to add vlan id 500 to my local > nics using vconfig but I still was n’t able to do access the CPVM or user > VMs. I guess my understanding of vlans with security groups is limited, so > if anyone knows about this feature or has used it - please help with some > regression testing. I plan to continue testing this week with VMWare and > fix any issues we find. > >>> > >>> > >>> ### Testing against Xen, KVM, VMWare etc. > >>> > >>> With a recent improvement to allow cpu features (such as vmx) on KVM > hosts ( > https://github.com/apache/cloudstack/commit/58cc569273905c50d089f9fd82fe80028b4e9775), > it’s now possible to run KVM, ESX, XenServer, OVM3 (LXC, and hopefully > others) on KVM (in case of ESX a patched qemu would be needed: > http://people.apache.org/~bhaisaab/qemu/). Such a (ansible based) tool > aims to be reproduce such an environment for anyone and serve as a > developer kit (not to be confused with the DevCloud appliance, will share > more details of the developer kit later) and as an infrastructure to run > integration testing suite we have now. This would allow us to > build/test/run CloudStack against various nested-virtualized hosts running > as guests on KVM, using CloudStack. > >>> > >>> Meanwhile, Abhi is focusing on existing (integration) tests and has > increased the number of TravisCI tests: > https://travis-ci.org/apache/cloudstack/builds/60365947 > >>> > >>> Our goal with these efforts is to (1) have a developer kit that a > developer can have locally to build/test/develop CloudStack, (2) scale it > up for bigger baremetal servers to run long running integration tests and > have a better QA automation. > >>> > >>> Regards, > >>> Rohit Yadav > >>> Software Architect, ShapeBlue > >>> M. +91 88 262 30892 | rohit.ya...@shapeblue.com > >>> Blog: bhaisaab.org | Twitter: @_bhaisaab > >>> > >>> > >>> > >>> Find out more about ShapeBlue and our range of CloudStack related > services > >>> > >>> IaaS Cloud Design & Build< > http://shapeblue.com/iaas-cloud-design-and-build//> > >>> CSForge – rapid IaaS deployment framework< > http://shapeblue.com/csforge/> > >>> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> > >>> CloudStack Software Engineering< > http://shapeblue.com/cloudstack-software-engineering/> > >>> CloudStack Infrastructure Support< > http://shapeblue.com/cloudstack-infrastructure-support/> > >>> CloudStack Bootcamp Training Courses< > http://shapeblue.com/cloudstack-training/> > >>> > >>> This email and any attachments to it may be confidential and are > intended solely for the use of the individual to whom it is addressed. Any > views or opinions expressed are solely those of the author and do not > necessarily represent those of Shape Blue Ltd or related companies. If you > are not the intended recipient of this email, you must neither take any > action based upon its contents, nor copy or show it to anyone. Please > contact the sender if you believe you have received this email in error. > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue > Services India LLP is a company incorporated in India and is operated under > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a > company incorporated in Brasil and is operated under license from Shape > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of > South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is > a registered trademark. > >> > > > > Regards, > > Rohit Yadav > > Software Architect, ShapeBlue > > M. +91 88 262 30892 | rohit.ya...@shapeblue.com > > Blog: bhaisaab.org | Twitter: @_bhaisaab > > > > > > > > Find out more about ShapeBlue and our range of CloudStack related > services > > > > IaaS Cloud Design & Build< > http://shapeblue.com/iaas-cloud-design-and-build//> > > CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/> > > CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/> > > CloudStack Software Engineering< > http://shapeblue.com/cloudstack-software-engineering/> > > CloudStack Infrastructure Support< > http://shapeblue.com/cloudstack-infrastructure-support/> > > CloudStack Bootcamp Training Courses< > http://shapeblue.com/cloudstack-training/> > > > > This email and any attachments to it may be confidential and are > intended solely for the use of the individual to whom it is addressed. Any > views or opinions expressed are solely those of the author and do not > necessarily represent those of Shape Blue Ltd or related companies. If you > are not the intended recipient of this email, you must neither take any > action based upon its contents, nor copy or show it to anyone. Please > contact the sender if you believe you have received this email in error. > Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue > Services India LLP is a company incorporated in India and is operated under > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a > company incorporated in Brasil and is operated under license from Shape > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of > South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is > a registered trademark. > -- *Mike Tutkowski* *Senior CloudStack Developer, SolidFire Inc.* e: mike.tutkow...@solidfire.com o: 303.746.7302 Advancing the way the world uses the cloud <http://solidfire.com/solution/overview/?video=play>*™*
