Thanks Marcus - I wanted to work through some concrete steps for the infra now that we have some idea what we want to do. We should have basic product setup and install and a single deployvm test for all possible configurations:
As a start can we work to setup devcloud and devcloud-kvm environments for jenkins? Every checkin should trigger a devcloud instance to spinup, start a server and run a simple deployvm test. This is done per-commit for the simulator but would be good to have it for devcloud as well. Can you help with devcloud-kvm? I can help with any jenkins configurations and the tests themselves. If someone would like to bring up a node for devcloud as well that would be great. It should help everyone understand how to add infrastructure to jenkins for tests. On Wed, Sep 11, 2013 at 11:17:42PM -0600, Mike Tutkowski wrote: > Yeah, I would love to get a SolidFire test suite up and running and plugged > into the main builds. > > > On Wed, Sep 11, 2013 at 11:15 PM, Marcus Sorensen <shadow...@gmail.com>wrote: > > > I think the test infra as described is great, but I think we're > > hurting a little more for basics. For example, we don't need a full > > infrastructure with hardware to ensure that the support matrix works. > > I could bring up a VM with CentOS and one with Ubuntu, and test NFS, > > CLVM, and RBD on each. CLVM just needs a volume group, NFS can be > > exported locally, and RBD can run on localhost node (ceph has how-tos > > for this to get your feet wet, that also buys us S3 compatible object > > storage for testing secondary). Two VMs with maybe 2 cores, 4GB ram > > each, and I think we could knock out a big swath of the basic "does it > > work on the supported platforms" that we're missing with a very simple > > automated testing. We can easily donate that much. > > > > I agree that third parties would need to plug in their own testing > > (solid fire, as you mention). And certainly testing full blown > > deployment from the ground up like it sounds like we are doing is > > great and necessary, I just want to plug a few holes and add some > > basic sanity checking that we seem to keep getting tripped up on. > > > > On Wed, Sep 11, 2013 at 10:23 PM, Prasanna Santhanam <t...@apache.org> > > wrote: > > > As Sebastien said, it's easy to get you the credentials for jenkins. > > > Anyone with commit rights can request for an account. In fact one is > > > created soon as you commit. I just need to adjust the credentials. > > > (We'll move to git based job configurations but later) > > > > > > Citrix is unable to test various configurations for lack of necessary > > > resources. for eg: It would be hard to test something that requires > > > hardware resources like Nicira/Midokura/Solidfire. The current testbed > > > is also limited in that it only deploys standard zone models. I have > > > only one storage node to spare on which NFS is configured. > > > > > > CloudStack can be deployed and configured in so many ways that I don't > > > think a single testbed cycling through all models is going to be > > > effective in testing every possible configuration in time. This is why > > > I'd like everyone of us to chip-in and use each others resources to > > > make the infrastructure better. > > > > > > The RBD store at least will require sometime for us to bring up. It > > > would be best if we could roll a few hosts from different datacenters > > > up into jenkins. Object storage backed CS with something like Riak is > > > another untested configuration. It is definitely tested within Citrix > > > Labs but those testbeds are internal and cannot be exposed to the > > > community. We've got corporate IT which wouldn't like that very much :) > > > > > > Ultimately, I'd want testbeds span across companies contributing to > > > cloudstack. I wouldn't want any single company X to hold the resources > > > and control allocation for testing even though that is not the case at > > > all. > > > > > > We still need to figure out how securely these deployments can be > > > brought into jenkins and who holds keys to the infrastructure. I'm no > > > secure conscious sysadmin so I'm hoping for inputs from operators > > > deploying cloudstack. > > > > > > On Wed, Sep 11, 2013 at 11:12:34AM -0600, Marcus Sorensen wrote: > > >> Again, I'm not knocking Citrix. If anything, the issue is that they > > tend to > > >> be so generous and community oriented that it surprises me when I find > > out > > >> that certain donation is limited to their interests. Its perfectly > > >> reasonable, e.g. my own donations are mostly limited to KVM. > > >> On Sep 11, 2013 10:52 AM, "Marcus Sorensen" <shadow...@gmail.com> > > wrote: > > >> > > >> > I do understand that. The email I received just triggered warning > > bells > > >> > because it gave me the impression that the QA team as it stands isn't > > >> > testing anything that Citrix doesn't care about, regardless of what > > the > > >> > community has put on the support matrix. This includes even basic > > configs > > >> > that the community claims to support like KVM on Ubuntu as the 4.1 > > release > > >> > shows, and other things that we may already have infra for but just > > haven't > > >> > implemented. > > >> > > > >> > That led me to wondering how much control the community really has > > over > > >> > testing. Its good to know that we can roll our own nodes up into > > Jenkins, > > >> > and/or modify tests if the infrastructure is already there. We just > > need to > > >> > raise awareness as a community that there are still holes in > > resources and > > >> > a need for donations to provide the minimum testing required for our > > >> > support matrix. I think David's email about release requirements is a > > good > > >> > step. > > >> > > > >> > If possible I'd like to modify the existing KVM testing to support > > testing > > >> > NFS, CLVM, and RBD. This can all be done with a single host (that > > >> > presumably already exists), we just need to set up the storage on the > > host > > >> > and add create pool commands and volume create/delete tests. I'll > > have to > > >> > figure out how to go about getting admin rights on the KVM test hosts > > to > > >> > configure the storage types or work with someone. If we can't do that > > due > > >> > to company logistics, I can easily stand up a VM or two to cover all > > of the > > >> > KVM mgmt/host hypervisor and storage configs if I can figure out how > > to > > >> > integrate. > > >> > On Sep 11, 2013 2:10 AM, "Sebastien Goasguen" <run...@gmail.com> > > wrote: > > >> > > > >> >> > > >> >> On Sep 11, 2013, at 3:02 AM, Prasanna Santhanam <t...@apache.org> > > wrote: > > >> >> > > >> >> > CloudStack API actions are agnostic of underlying infrastructure > > and > > >> >> > most cases can fall into such a category as you describe. But > > imagine > > >> >> > this - I want to test snapshots .. > > >> >> > > > >> >> > so i take a snapshot and verify if it backedup correctly against a > > >> >> > ceph object store, nfs store or iscsi store. that sort of test is > > >> >> > going to involve more than just api actions. > > >> >> > > > >> >> > or say - i want to test multiple shared networks a VM gets deployed > > >> >> > into. Do I assume the deployment has multiple shared networks? Can > > i > > >> >> > add my own network into the deployment? > > >> >> > > > >> >> > or even - I want to exhaust all the public network IPs and check if > > >> >> > the next deployed VM picks up an IP in the new public range I've > > >> >> > added. This sort of test assumes that all the necessary networking > > is > > >> >> > in place and also hurts VM deployments of all tests that run at the > > >> >> > same time. > > >> >> > > > >> >> > It's a difficult balance to strike but we have to begin somewhere. > > >> >> > Start with the basic minimum that every infra can run. infra > > specifc > > >> >> > tests skip if thing are unsuitable, but will run for someone who > > wants > > >> >> > to test that feature > > >> >> > > >> >> A small point here to make is that jenkins.cloudstack.org is open to > > >> >> anyone. > > >> >> Prasanna has created an account for me and I am (slowly) working on > > >> >> adding tests for clients including aws. > > >> >> > > >> >> Anyone could use this jenkins instance, bring in slaves from "home" > > and > > >> >> setup tests? > > >> >> > > >> >> Back to the solidfire example, I think Mike could easily contribute > > one > > >> >> node that has a solidfire storage, then contribute Marvin tests that > > would > > >> >> run on jenkins.c.o and target his slave specifically. Same for KVM on > > >> >> Ubuntu... > > >> >> > > >> >> -sebastien > > >> >> > > >> >> > > >> >> > > > >> >> > On Tue, Sep 10, 2013 at 11:53:15PM -0700, Ahmad Emneina wrote: > > >> >> >> That's a good question, I'm not sure how preconditions work with > > >> >> >> Marvin cases, but I know the tests are run generically. Say I run > > >> >> >> copyvolumeToPrimary (not sure this test exists, hypothetical at > > the > > >> >> >> moment), it gets run against a slew of infrastructure > > configurations > > >> >> >> using local storage as well as shared (NSF, iscsi, ceph...) back > > >> >> >> ends. So just dropping my test into a storage suite should give it > > >> >> >> some guarantee its hitting a few different storage back-ends. > > That's > > >> >> >> how i understand it works today, I'll defer to Prasanna or > > Sudha... > > >> >> >> Or anyone else that runs tests aggressively to fill in the gaps > > and > > >> >> >> make corrections. > > >> >> >> > > >> >> >> Ahmad > > >> >> >> > > >> >> >> On Sep 10, 2013, at 11:43 PM, Marcus Sorensen < > > shadow...@gmail.com> > > >> >> wrote: > > >> >> >> > > >> >> >>> But if the test requires some sort of preconfiguration, what then > > >> >> (e.g. test NFS primary storage would need a local or remote NFS > > >> >> configured)? do I need to roll my own, or can I touch the existing > > test > > >> >> infra and do the preconfigure? > > >> >> >>> > > >> >> >>> On Sep 11, 2013 12:34 AM, "Prasanna Santhanam" <t...@apache.org> > > >> >> wrote: > > >> >> >>>> Yes - Once your test goes into the repo, it should get picked > > in the > > >> >> subsequent > > >> >> >>>> run. > > >> >> >>>> > > >> >> >>>> Jenkins installations from various companies can be combined > > into a > > >> >> single > > >> >> >>>> landing page. Jenkins itself doesn't support master/slave but it > > >> >> does through > > >> >> >>>> the gearman plugin. It's something I have tried using with VMs > > but > > >> >> not with > > >> >> >>>> real infra - but it is entirely possible. > > >> >> >>>> > > >> >> >>>> On Tue, Sep 10, 2013 at 11:17:53PM -0700, Ahmad Emneina wrote: > > >> >> >>>>> I think there are jenkins slaves that run the nicera plugins > > on/at > > >> >> Schuberg > > >> >> >>>>> Philis housed infrastructure. The Citrix jenkins nodes also > > runs as > > >> >> slaves > > >> >> >>>>> that connect back to the apache owned/controlled jenkins. No > > reason > > >> >> why > > >> >> >>>>> testing infra need be so consolidated, it just so happens no > > one is > > >> >> putting > > >> >> >>>>> their hardware where their mouth is. > > >> >> >>>>> > > >> >> >>>>> I also assume if your marvin tests get accepted upstream, > > they'll be > > >> >> >>>>> included in the nightly runs/reports. Prasanna correct me if > > I'm > > >> >> wrong. > > >> >> >>>>> > > >> >> >>>>> > > >> >> >>>>> On Tue, Sep 10, 2013 at 11:02 PM, Marcus Sorensen < > > >> >> shadow...@gmail.com>wrote: > > >> >> >>>>> > > >> >> >>>>>> CloudStack Dev, > > >> >> >>>>>> I was emailed about some of the testing questions I > > brought up > > >> >> >>>>>> over the last few threads, and a few things were pointed out > > to me > > >> >> >>>>>> that I think we should try to remedy. Primarily, that the > > testing > > >> >> >>>>>> environment is owned by Citrix, the QA team is primarily > > >> >> Citrix-run, > > >> >> >>>>>> and the testing done is focused on the use models that Citrix > > >> >> >>>>>> develops. > > >> >> >>>>>> I've been assured that the test infrastructure is for > > everyone, > > >> >> >>>>>> and I'm not at all trying to say that there's a problem with > > Citrix > > >> >> >>>>>> focusing their work on their own interests, but I'm not sure > > that > > >> >> >>>>>> anyone outside of Citrix really knows how to add their own > > stuff to > > >> >> >>>>>> this testing infrastructure (perhaps for lack of trying, I > > don't > > >> >> >>>>>> know). > > >> >> >>>>>> I haven't really put together enough thought to know how to > > >> >> tackle > > >> >> >>>>>> this, but my gut tells me that we need some sort of > > community-owned > > >> >> >>>>>> testing roll-up, where everyone can do their own testing in > > >> >> whatever > > >> >> >>>>>> infrastructure and submit hourly, daily, weekly results. If > > my test > > >> >> >>>>>> fits into the Citrix test infrastructure and I can figure out > > how > > >> >> to > > >> >> >>>>>> get it there, great. If not, I can roll my own and integrate > > it via > > >> >> >>>>>> some API. For example the SolidFire guys may wan to run > > automated > > >> >> >>>>>> regression testing. That probably won't be doable in the > > Citrix > > >> >> >>>>>> infrastructure, but they may want to script a daily > > >> >> >>>>>> git-pull/build/deploy zone/create volume and it seems logical > > that > > >> >> >>>>>> we'd want to support it. > > >> >> >>>>>> Thoughts? Anyone have experience with such things? Can we > > have a > > >> >> >>>>>> master/slave scenario with Jenkins? Perhaps the Citrix > > environment > > >> >> >>>>>> already supports something like this via Jenkins API? > > >> >> >>>>>> > > >> >> >>>> > > >> >> >>>> -- > > >> >> >>>> Prasanna., > > >> >> >>>> > > >> >> >>>> ------------------------ > > >> >> >>>> Powered by BigRock.com > > >> >> > > > >> >> > -- > > >> >> > Prasanna., > > >> >> > > > >> >> > ------------------------ > > >> >> > Powered by BigRock.com > > >> >> > > > >> >> > > >> >> > > > > > > -- > > > Prasanna., > > > > > > ------------------------ > > > Powered by BigRock.com > > > > > > > > > -- > *Mike Tutkowski* > *Senior CloudStack Developer, SolidFire Inc.* > e: mike.tutkow...@solidfire.com > o: 303.746.7302 > Advancing the way the world uses the > cloud<http://solidfire.com/solution/overview/?video=play> > *?* -- Prasanna., ------------------------ Powered by BigRock.com
