> -----Original Message----- > From: prasanna [mailto:srivatsav.prasa...@gmail.com] On Behalf Of > Prasanna Santhanam > Sent: Sunday, March 10, 2013 8:49 AM > To: CloudStack Dev > Subject: Re: [CI] marvin tests on jenkins > > Update: > > The setup is now reliably functional with xen and kvm hosts coming up > successfully, mgmt servers spinning up fine, server health is also good with > systemvms up and functional. A misbehaving network issue prevented us > from turning on the infra. This is now worked around: > > All the sequence of jobs are on: > http://jenkins.cloudstack.org/view/cloudstack-qa/ > > (Currently set to test against the 4.1 branch. The packaging has been broken > for a couple of days but the setup will use the > lastSuccessfulBuild) > > test-matrix (master job that triggers all others, xen, kvm profiles are > running) > - test-packaging (latest package tested from the s3 repo at > yum.cloudstack.org) > - test-environment-refresh (xen/kvm hosts are reimaged and agent package > is seeded) (Above two run in parallel) > - test-setup-advanced-zone (brings up an advanced zone) > - test-cloudstack-smoke (right now server health is checked, but will be > adding the actual tests tomorrow) Thanks!!! It's a big step forward to get CI work. I have few questions regarding to marvin test though: Where do we put the marvin unit test cases? I can't find them on master branch. I am trying to bring up the marvin test against simulator and devcloud(I think the smoke test already did that), we may need just one maven command to setup mgt server and simulator on developer's environment, and one maven command to run marvin test cases against simulator. The benefits of using simulator: 1. Easy to setup 2. Easy to scale, ten of thousands hypervisor hosts and VMs are not a problem at all. 3. Easy to inject errors in simulator(it should be, there is API to inject errors in simulator resource code) Developers should test against simulator before test your features and bug fixes on real hardware, if the features and bug fixes are not related to specific hardware. The challenges we have(may not specific to test against simulator): 1. How to write a test case regardless of the test environment? The same test case should be work with different test setup, no matter it's simulator or xenserver, vmware etc. 2. How to write a test case just specific to a test environment and let the test framework know the requirement? We have so many test environment combinations, e.g. advanced zone, advanced zone with all complicated network setup, basic zone, multiple clusters, multiple zones etc, etc. We may need to standardize all the test environment combinations, category them, so the test case can be categorized to one specific test environment. 3. How to make write marvin test easier for developer and QA? I know you are working on it on the marvin refactor branch, do you have a target, what you are trying to achieve in short term or long term? Or what's the feedbacks/complains from people actually writing marvin unit test before? Not sure above issues are already fixed or not, because I haven't look at the current existing test cases yet... If not, let's fix them.
> > Most of the work has been happening on github [1] [2] making it less visible > on the lists. My fault entirely. I know this might have caused some > impatience among folks. But hopefully we can pull off 100% pass rate before > 4.1 goes out the door! > > [1] https://github.com/vogxn/acs-infra-test/tree/4.1 > [2] https://github.com/vogxn/cloud-autodeploy/tree/acs-infra-fmt > > Thanks, > > -- > Prasanna., > > On Fri, Feb 22, 2013 at 08:17:49PM +0530, Prasanna Santhanam wrote: > > The build pipeline for running marvin tests on physical hardware is > > now finally shaping up. The test-matrix [1] runs through a few valid > > configurations across hypervisors and distributions. Please have a > > look. > > > > Currently the RPM package is tested with a centos63 distro. I'll add > > ubuntu as well when the packaging is sorted. The hypervisors enabled > > are the ones in the OSS package (kvm and xen). > > > > test-matrix splits into a few jobs after choosing a hypervisor and a > > distro for the management server > > - test-packaging (installs the latest packages from > > yum.cloudstack.org) > > - test-environment-refresh (cleans up the hypervisors to prepare a > > fresh run) > > - test-setup-advanced-zone (brings together an advanced zone, TODO: > > basic zone deployments) > > - test-cloudstack-smoke (disabled for now until above jobs > > stabilize, will run the existing smoke tests) > > > > All the end-to-end configuration is automated using puppet recipes and > > there is negligible manual interference at this point. Except to > > collect logs, which I'll push on to the jenkins artifacts too. > > > > The plan is to have functional tests hitting the repo run immediately > > on the existing configuration. As well as run tests on a regular > > schedule. I'll put together the schedule - a complete regression suite > > twice a week and a smoke test every few hours would be the goal I'd > > like to begin with. > > > > The tests being dated will likely have failures and I'll start fixing > > them starting early next week. Patches and fixes welcome. > > > > [1] http://jenkins.cloudstack.org/view/cloudstack-qa/job/test-matrix/ > > > > Thanks, > > > > -- > > Prasanna.,
