> -----Original Message-----
> From: Pierre-Luc Dion [mailto:pd...@cloudops.com]
> Sent: Wednesday, August 20, 2014 11:20 AM
> To: dev@cloudstack.apache.org; Will Stevens
> Subject: [DISCUSSION] CI
>
> I'd like to move this discussion out of the GIT workflow thread.
>
> Do we have any CI strategy in place or a starting point at least?
TravisCI will be a starting point for now. It tests against simulator only.
I am working on a solution to test against KVM(that's most of users are
concerned about) in a machine I created in
Public cloud. It has big memory, super fast disk IO, should be a good solution
for KVM test. It may take few days to get it work.
They are both for simple test, and in small scale.
>
> I've never worked with marvin I from the small understanding that I have
> from it, it might not be the right too. I don't know.
>
> I've spend some times recently with test-kitchen and serverspec and found
> that quite promising as test platform for "real cloudstack deployment test
> scenarios". By using test framework like Rspec the test output is clean to
> read for human and test scenarios seams quite simple to write.
>
> I've started a rspec lib for cloudstack to enhance serverspec for CloudStack
> tests (cloudstack-rspec). I'm wondering if it would be usable in future CI
> system for CloudStack...
I am familiar with Specs, like Rspec, but in scala. I agree with you that the
BDD style test case
Is easier to read/write. It's good to have, but I think before we move to new
test framework, we should figure out the issues
We have today in Marvin, so that we can take effort to address them, either
fixing Marvin, or switch to a new test framework.
The issue I have in current Marvin, is that it still needs to write a lot of
setup code for each test case, which is complicated enough. Let's take an
Marvin test case as an example:
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=blob;f=test/integration/smoke/test_vm_life_cycle.py;h=57e47c24838ffc2ec5e9af4ad3872d5e27342d91;hb=HEAD
Like this kind of code in test case setup code:
# Get Zone, Domain and templates
cls.domain = get_domain(cls.apiclient)
cls.zone = get_zone(cls.apiclient, testClient.getZoneForTests())
cls.services['mode'] = cls.zone.networktype
#If local storage is enabled, alter the offerings to use localstorage
#this step is needed for devcloud
if cls.zone.localstorageenabled == True:
cls.services["service_offerings"]["tiny"]["storagetype"] = 'local'
cls.services["service_offerings"]["small"]["storagetype"] = 'local'
cls.services["service_offerings"]["medium"]["storagetype"] = 'local'
this kind of information should be provided by test framework automatically,
instead of explicitly set in every test case.
Do you have other concerns about current Marvin, other than test style?
>
> Also, having a CI in place for cloudstack is quite of a huge deal if we want
> to
> validate all kind of deployment scenario, networks, hypervisor version and
> guest OS version.
Can't agree with you more.
>
>
> For those that have some knowledge on the CI subject in CloudStack please
> update the community. I don't feel alone being lost about this CloudStack
> topic.
There is an update-to-date write up regarding to Marvin:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Marvin+-+Testing+with+Python
Marvin does two things, one is to call cloudstack mgt API to add resources into
CloudStack mgt server, such as hypervisors, primary storages, networks, and
secondary storages etc.
Another one is to run test cases under tools/integration/smoke or any other
python test cases.
It doesn't cover how to deploy hypervisors on real HW, how to setup primary
storages, secondary storage etc. That's something like test-kitchen and
serverspec can help, I think.
>
> Thanks,
>
>
> Pierre-Luc
