> -----Original Message----- > From: Pierre-Luc Dion [mailto:pd...@cloudops.com] > Sent: Thursday, August 21, 2014 5:06 AM > To: dev@cloudstack.apache.org > Cc: Will Stevens > Subject: Re: [DISCUSSION] CI > > Thanks for you responses, > > The revamp of marvin look promising, I'll have a closer look at it, even if > I'm > not Python fan ;). In our side, we have close to fully automated installation > of CloudStack, XenServer hypervisor and an initial zone creation. Our missing
Will it be open sourced?:) > part is automated test cases to use it. So we could provide infrastructure and > run tests against it for one or more test scenarios during QA phase of > release. > Hopefully this could be integrated to the CI. Currently http://jenkins.buildacloud.org/, is CloudStack CI portal. If you can provide Infrastructure to integrate with it, that will be awesome! > > Would it make sense to have a sheet, maybe a wiki page that list all > infrastructure test scenarios and their result per version of ACS? > for: > xenserver, hyperv, vmware,kvm > advanced, simplenetworking, vlan/vxlan,gre isolated, networks s3,swift,nfs, > storage plugin > > Unless we have this kind of results somewhere, I don't feel we really keep > track of what has been tested on each release. It's a good idea. Let me ask the team who wrote these test cases, maybe they have answers. > > > > > *Pierre-Luc DION* > Architecte de Solution Cloud | Cloud Solutions Architect t 855.652.5683 > > *CloudOps* Votre partenaire infonuagique* | *Cloud Solutions Experts > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw > @CloudOps_ > > > > On Thu, Aug 21, 2014 at 2:03 AM, Prasanna Santhanam <t...@qubole.com> > wrote: > > > I use rspec in Ruby these days and find it get complex to maintain > > pretty quickly. It's good for dealing with REST apis and middleware > > but with Marvin (and python in general) I find infrastructure handling > > easier. Stack traces of BDD code is also a little bit hard to > > troubleshoot and comprehend. I may be biased because my Python > > expertise is better, so you can take this with a pinch of salt. > > > > Edison is right however, the setup in marvin is atrociously complex > > and needs to be a lot simpler and intuitive. In most cases I don't > > want to specify tons of args (service offering, disk offering, network > > type) to create a simple vM that is part of every test. To this end I > > refactored Marvin to use easily defined factories (in lines with > > Ruby's factory_girl) and also demo-ed some code in the branch. Haven't > > had time to maintain and revamp this honestly. So if anyone wants to > > pick up from there, I'm happy to help. > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+Refactor > > > > > > > > On Thu, Aug 21, 2014 at 4:30 AM, Edison Su <edison...@citrix.com> wrote: > > > > > > > > >> -----Original Message----- > > >> From: Pierre-Luc Dion [mailto:pd...@cloudops.com] > > >> Sent: Wednesday, August 20, 2014 11:20 AM > > >> To: dev@cloudstack.apache.org; Will Stevens > > >> Subject: [DISCUSSION] CI > > >> > > >> I'd like to move this discussion out of the GIT workflow thread. > > >> > > >> Do we have any CI strategy in place or a starting point at least? > > > TravisCI will be a starting point for now. It tests against > > > simulator > > only. > > > I am working on a solution to test against KVM(that's most of users > > > are > > concerned about) in a machine I created in > > > Public cloud. It has big memory, super fast disk IO, should be a > > > good > > solution for KVM test. It may take few days to get it work. > > > They are both for simple test, and in small scale. > > >> > > >> I've never worked with marvin I from the small understanding that I > > >> have from it, it might not be the right too. I don't know. > > >> > > >> I've spend some times recently with test-kitchen and serverspec and > > found > > >> that quite promising as test platform for "real cloudstack > > >> deployment > > test > > >> scenarios". By using test framework like Rspec the test output is > > >> clean > > to > > >> read for human and test scenarios seams quite simple to write. > > >> > > >> I've started a rspec lib for cloudstack to enhance serverspec for > > CloudStack > > >> tests (cloudstack-rspec). I'm wondering if it would be usable in > > future CI > > >> system for CloudStack... > > > I am familiar with Specs, like Rspec, but in scala. I agree with > > > you > > that the BDD style test case > > > Is easier to read/write. It's good to have, but I think before we > > > move > > to new test framework, we should figure out the issues > > > We have today in Marvin, so that we can take effort to address them, > > either fixing Marvin, or switch to a new test framework. > > > The issue I have in current Marvin, is that it still needs to write > > > a > > lot of setup code for each test case, which is complicated enough. > > Let's take an Marvin test case as an example: > > https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blob;f=test > > > /integration/smoke/test_vm_life_cycle.py;h=57e47c24838ffc2ec5e9af4ad38 > > 72d5e27342d91;hb=HEAD > > > > > > Like this kind of code in test case setup code: > > > > > > # Get Zone, Domain and templates > > > cls.domain = get_domain(cls.apiclient) > > > cls.zone = get_zone(cls.apiclient, testClient.getZoneForTests()) > > > cls.services['mode'] = cls.zone.networktype > > > > > > #If local storage is enabled, alter the offerings to use > > localstorage > > > #this step is needed for devcloud > > > if cls.zone.localstorageenabled == True: > > > cls.services["service_offerings"]["tiny"]["storagetype"] > > > = > > 'local' > > > > > > cls.services["service_offerings"]["small"]["storagetype"] = > > 'local' > > > > > > cls.services["service_offerings"]["medium"]["storagetype"] = > > 'local' > > > > > > this kind of information should be provided by test framework > > automatically, instead of explicitly set in every test case. > > > Do you have other concerns about current Marvin, other than test style? > > > > > > > > >> > > >> Also, having a CI in place for cloudstack is quite of a huge deal > > >> if we > > want to > > >> validate all kind of deployment scenario, networks, hypervisor > > >> version > > and > > >> guest OS version. > > > > > > > > > Can't agree with you more. > > > > > >> > > >> > > >> For those that have some knowledge on the CI subject in CloudStack > > please > > >> update the community. I don't feel alone being lost about this > > CloudStack > > >> topic. > > > > > > There is an update-to-date write up regarding to Marvin: > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+- > +Testin > > g+with+Python > > > Marvin does two things, one is to call cloudstack mgt API to add > > resources into CloudStack mgt server, such as hypervisors, primary > > storages, networks, and secondary storages etc. > > > Another one is to run test cases under tools/integration/smoke or > > > any > > other python test cases. > > > It doesn't cover how to deploy hypervisors on real HW, how to setup > > primary storages, secondary storage etc. That's something like > > test-kitchen and serverspec can help, I think. > > > > > >> > > >> Thanks, > > >> > > >> > > >> Pierre-Luc > > > > > > > > -- > > Prasanna., > >
