On Thu, Jan 31, 2013 at 12:59:27PM +0530, Ahmad Emneina wrote:
> Can we not group the tests vs standardizing all test to work against
> devcloud? that way one can run the devcloud test suite against devcloud and
> not be bothered with failing advanced zone tests.
Yeah - they should be grouped and tagged seperately for different
environments. @attr(tags='devcloud) tests are only for devcloud. Some
can run across multiple environments and are tagged that way.
>
> On Wed, Jan 30, 2013 at 11:03 PM, Marcus Sorensen <shadow...@gmail.com>wrote:
>
> > On the integration testing, my impressions are that they're written
> > against a specific environment. That's probably expected, being an
> > actual functionality test, but it makes it hard to actually run them.
> > Some of them work against devcloud in a basic zone, others work for an
> > advanced setup, some tests don't seem to work at all and I'm not sure
> > if they're just unfinished or specific to someone's test environment.
> > Everyone might be writing tests but they only work in their
> > environment.
You're right. devcloud style testing wasn't the intention when the
tests were written since devcloud came later. The expected functional
test setup involved the marvin config I shared via github [1]. The
last run on 4.0 had the summary as shown on jenkins [2]. Another
reason for the failures is that there are fixes waiting to be merged
into the asf repo from the contributors. IP clearance on contributed
tests [3] has left us stuck in merging the fixes to ASF that were
supposed to come in through reviewboard. Some grouping in the tests
exists to isolate the expected setup and run the test appropriately. I
explain this in the tutorial on testing [4]. Yet another set of
failures is because we adjust test data in the Services() dictionaries
to fit the setup. This usually involves credential information. But
those aren't too many.
> >
> > I've mentioned this before, but it should be possible to standardize
> > on a marvin config that is self-contained (management server inside),
> > that anyone can download and run the tests on, write tests against.
> > Even possibly have a devcloud vm that pulls down master on a cron job,
> > runs the tests, and emails/reports the results every day. It should
> > even be possible to have a devcloud and devcloud-kvm with the same
> > configs (same address ranges, etc) so the tests can be run against
> > both.
> >
We do have a devcloud CI setup for the purpose of running only
devcloud integration tests in a continuous fashion [5]. It's not
perfect but it's mostly consistent. There are some TODOs to address.
[6] devcloud-kvm is one of them. With such a setup running, test
writers and devs can get constant feedback on how their code is
behaving after checkin and acts as a first level of defense.
We should certainly standardize on commonly deployed setups. Not
everyone deploys cloudstack the same way so I expect the following to
cover the general use cases.
1. advanced zone - with hosts in a cluster, hosts in diff't pods,
hosts across zones
- can test inter-cluster and inter-zone
- can test high availability
2. basic zone w/o and w/ security groups
3. advanced zone with netscalers/other lb elements
4. some setup with object store
5. EIP/ELB style deployments (low priority)
I documented some of the work for the live test infrastructure but got
engaged into moving it before I could finish the documentation. It is
a draft at this point - somewhat useable. [7]
However, I don't have enough hardware to run and recycle my setups
across all these configs. So hoping others will pitch in with some
setups for connecting with the CI. I'm more than happy to help
integrate them. When we run the component tests they take up a good
~8-10h to run. So some more hardware will be helpful.
[1] https://github.com/vogxn/cloud-autodeploy/blob/acs-infra-test/xen.cfg
[2]
http://jenkins.cloudstack.org/view/debug/job/nightly-smoke-xen/102/testReport/
[3] http://comments.gmane.org/gmane.comp.apache.cloudstack.devel/8142
[4] section on attributes:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Testing+with+Python#TestingwithPython-MarvinNosePlugin
[5]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Devcloud+Continuous+Tests
[6]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Devcloud+Continuous+Tests#DevcloudContinuousTests-TODOsandFIXMEs
[7]
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Live+test+infrastructure
--
Prasanna.,
