On Thu, Aug 21, 2014 at 1:32 PM, Edison Su <edison...@citrix.com> wrote:
> > > > -----Original Message----- > > From: Pierre-Luc Dion [mailto:pd...@cloudops.com] > > Sent: Thursday, August 21, 2014 5:06 AM > > To: dev@cloudstack.apache.org > > Cc: Will Stevens > > Subject: Re: [DISCUSSION] CI > > > > Thanks for you responses, > > > > The revamp of marvin look promising, I'll have a closer look at it, even > if I'm > > not Python fan ;). In our side, we have close to fully automated > installation > > of CloudStack, XenServer hypervisor and an initial zone creation. Our > missing > > > Will it be open sourced?:) Of course :-) for the Cloudstack configuration part. https://github.com/cloudops/cookbook_cloudstack For the XenServer side it's a bit arch because we basically manage xenserver kickstart with a Chef Cookbook that is quite ugly. but could be opensourced. Zone, service offerings, templates creation, I have a set of scripts not on git hub yet (still too ugly) but marven does it too. > > part is automated test cases to use it. So we could provide > infrastructure and > > run tests against it for one or more test scenarios during QA phase of > release. > > Hopefully this could be integrated to the CI. > Currently http://jenkins.buildacloud.org/, is CloudStack CI portal. If > you can provide > Infrastructure to integrate with it, that will be awesome! > > > > > Would it make sense to have a sheet, maybe a wiki page that list all > > infrastructure test scenarios and their result per version of ACS? > > for: > > xenserver, hyperv, vmware,kvm > > advanced, simplenetworking, vlan/vxlan,gre isolated, networks > s3,swift,nfs, > > storage plugin > > > > Unless we have this kind of results somewhere, I don't feel we really > keep > > track of what has been tested on each release. > It's a good idea. Let me ask the team who wrote these test cases, maybe > they have answers. > > > > > > > > > > > *Pierre-Luc DION* > > Architecte de Solution Cloud | Cloud Solutions Architect t 855.652.5683 > > > > *CloudOps* Votre partenaire infonuagique* | *Cloud Solutions Experts > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* tw > > @CloudOps_ > > > > > > > > On Thu, Aug 21, 2014 at 2:03 AM, Prasanna Santhanam <t...@qubole.com> > > wrote: > > > > > I use rspec in Ruby these days and find it get complex to maintain > > > pretty quickly. It's good for dealing with REST apis and middleware > > > but with Marvin (and python in general) I find infrastructure handling > > > easier. Stack traces of BDD code is also a little bit hard to > > > troubleshoot and comprehend. I may be biased because my Python > > > expertise is better, so you can take this with a pinch of salt. > > > > > > Edison is right however, the setup in marvin is atrociously complex > > > and needs to be a lot simpler and intuitive. In most cases I don't > > > want to specify tons of args (service offering, disk offering, network > > > type) to create a simple vM that is part of every test. To this end I > > > refactored Marvin to use easily defined factories (in lines with > > > Ruby's factory_girl) and also demo-ed some code in the branch. Haven't > > > had time to maintain and revamp this honestly. So if anyone wants to > > > pick up from there, I'm happy to help. > > > > > > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+Refactor > > > > > > > > > > > > On Thu, Aug 21, 2014 at 4:30 AM, Edison Su <edison...@citrix.com> > wrote: > > > > > > > > > > > >> -----Original Message----- > > > >> From: Pierre-Luc Dion [mailto:pd...@cloudops.com] > > > >> Sent: Wednesday, August 20, 2014 11:20 AM > > > >> To: dev@cloudstack.apache.org; Will Stevens > > > >> Subject: [DISCUSSION] CI > > > >> > > > >> I'd like to move this discussion out of the GIT workflow thread. > > > >> > > > >> Do we have any CI strategy in place or a starting point at least? > > > > TravisCI will be a starting point for now. It tests against > > > > simulator > > > only. > > > > I am working on a solution to test against KVM(that's most of users > > > > are > > > concerned about) in a machine I created in > > > > Public cloud. It has big memory, super fast disk IO, should be a > > > > good > > > solution for KVM test. It may take few days to get it work. > > > > They are both for simple test, and in small scale. > > > >> > > > >> I've never worked with marvin I from the small understanding that I > > > >> have from it, it might not be the right too. I don't know. > > > >> > > > >> I've spend some times recently with test-kitchen and serverspec and > > > found > > > >> that quite promising as test platform for "real cloudstack > > > >> deployment > > > test > > > >> scenarios". By using test framework like Rspec the test output is > > > >> clean > > > to > > > >> read for human and test scenarios seams quite simple to write. > > > >> > > > >> I've started a rspec lib for cloudstack to enhance serverspec for > > > CloudStack > > > >> tests (cloudstack-rspec). I'm wondering if it would be usable in > > > future CI > > > >> system for CloudStack... > > > > I am familiar with Specs, like Rspec, but in scala. I agree with > > > > you > > > that the BDD style test case > > > > Is easier to read/write. It's good to have, but I think before we > > > > move > > > to new test framework, we should figure out the issues > > > > We have today in Marvin, so that we can take effort to address them, > > > either fixing Marvin, or switch to a new test framework. > > > > The issue I have in current Marvin, is that it still needs to write > > > > a > > > lot of setup code for each test case, which is complicated enough. > > > Let's take an Marvin test case as an example: > > > https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blob;f=test > > > > > /integration/smoke/test_vm_life_cycle.py;h=57e47c24838ffc2ec5e9af4ad38 > > > 72d5e27342d91;hb=HEAD > > > > > > > > Like this kind of code in test case setup code: > > > > > > > > # Get Zone, Domain and templates > > > > cls.domain = get_domain(cls.apiclient) > > > > cls.zone = get_zone(cls.apiclient, > testClient.getZoneForTests()) > > > > cls.services['mode'] = cls.zone.networktype > > > > > > > > #If local storage is enabled, alter the offerings to use > > > localstorage > > > > #this step is needed for devcloud > > > > if cls.zone.localstorageenabled == True: > > > > cls.services["service_offerings"]["tiny"]["storagetype"] > > > > = > > > 'local' > > > > > > > > cls.services["service_offerings"]["small"]["storagetype"] = > > > 'local' > > > > > > > > cls.services["service_offerings"]["medium"]["storagetype"] = > > > 'local' > > > > > > > > this kind of information should be provided by test framework > > > automatically, instead of explicitly set in every test case. > > > > Do you have other concerns about current Marvin, other than test > style? > > > > > > > > > > > >> > > > >> Also, having a CI in place for cloudstack is quite of a huge deal > > > >> if we > > > want to > > > >> validate all kind of deployment scenario, networks, hypervisor > > > >> version > > > and > > > >> guest OS version. > > > > > > > > > > > > Can't agree with you more. > > > > > > > >> > > > >> > > > >> For those that have some knowledge on the CI subject in CloudStack > > > please > > > >> update the community. I don't feel alone being lost about this > > > CloudStack > > > >> topic. > > > > > > > > There is an update-to-date write up regarding to Marvin: > > > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+- > > +Testin > > > g+with+Python > > > > Marvin does two things, one is to call cloudstack mgt API to add > > > resources into CloudStack mgt server, such as hypervisors, primary > > > storages, networks, and secondary storages etc. > > > > Another one is to run test cases under tools/integration/smoke or > > > > any > > > other python test cases. > > > > It doesn't cover how to deploy hypervisors on real HW, how to setup > > > primary storages, secondary storage etc. That's something like > > > test-kitchen and serverspec can help, I think. > > > > > > > >> > > > >> Thanks, > > > >> > > > >> > > > >> Pierre-Luc > > > > > > > > > > > > -- > > > Prasanna., > > > >
