On 06/07/2011 02:36 PM, Dan Prince wrote: > Hi Monty, > > I spent a few moments to put together some more detailed notes on > SmokeStack: > > http://wiki.openstack.org/smokestack > > Screenshots and a link to the live server are on that wiki page. > > At this point the ability to specify custom nova and glance branches > is supported. Each job runs a set of smoke tests that cover both the > OS and EC2 APIs.
Excellent. I'm looking forward to working with you to figure out how we can integrate this in a way which is usable/consumable by the Jenkins system. > We've been torpedoing quite a few branches with SmokeStack lately. > Its proving very useful in doing reviews and helping make sure we get > things right before they land in trunk. Yup. I couldn't agree more - the more pre-merge testing that can happen the better! Monty > -----Original Message----- From: "Monty Taylor" > <mord...@inaugust.com> Sent: Tuesday, June 7, 2011 12:30pm To: > "openstack@lists.launchpad.net" <openstack@lists.launchpad.net> Cc: > "Dan Prince" <dan.pri...@rackspace.com>, "Mihai Ibanescu" > <m...@rpath.com>, "Peter J. Pouliot" <ppoul...@suse.com> Subject: > Overview of CI/Testing > > Hey everybody, > > Here's a quick write up on what we have at the moment and what the > plan is moving forward, as it stands today. In case you weren't > aware, the Jenkins instance sits at: > > http://jenkins.openstack.org > > If you think you want to help administer/hack on our Jenkins, that > is managed through the openstack-ci-admins team on Launchpad.. > > As most of you are aware, the main trunk branches for nova, swift, > glance and burrow are all managed by Jenkins and Tarmac. What this > means in short is that jenkins/tarmac finds approved merge proposals, > runs the in-tree unit tests on them, and if successful, pushes back > to trunk. Additionally, we have jenkins jobs which run pep8, pylint > and code coverage jobs and produce reports. For instance: > > http://jenkins.openstack.org/job/nova-coverage/815/cobertura/? > > Once the code has been merged, we have jenkins jobs which produce > tarballs, and debian source packages which are pushed up to > launchpad PPAs so that effectively every trunk commit winds up having > an associated package. > > There is also a feature that it seems no one knew about which allows > developers to submit branch URLs to jenkins to have it run its tests > on that branch. For each project, this is ${project_name}-param, so > for instance: > > http://jenkins.openstack..org/job/nova-param/ > > Will allow anyone in the ~nova team to submit a branch and have > jenkins pull it and run on it what it would do via tarmac - helpful > when fixing something tarmac complained about. > > We are planning to get rid of tarmac and get that integrated in as a > Jenkins plugin, because otherwise supporting things like NTT's wish > for stricter coverage metrics as branch gatekeepers, or smoketesting > is preventing branches from getting in become really baroque to > support. > > ** If you like hacking on Java - this should be a fun little > project. Send me a note and I can outline what I was going to do and > if you want it it's all yours. ** > > Moving forward, the big ticket item is testing not only in-tree unit > tests, but installing things and testing that. > > Currently, we have jenkins spinning up a cloud server, copying the > most recently built debs from the last successful build to that > server, installing them, starting things up and then running the > smoketests against the installed code. Currently the install/setup is > being done by hand rather than via chef purely so that I could walk > through what's actually needed to get a single-node minimal install > working and collect information on bugs/workarounds that have to > happen. > > http://jenkins.openstack.org/job/nova-smoketests > > The test failures here are due to config issues which should go away > once I migrate that job to using chef for the node code - so I > expect that to go green soon. > > The next step then is to replace the use of shell commands over ssh > with the chef recipes. (now that I have a good handle on what's > going on/needed there, following the existing chef automation work is > much more fruitful) > > After that, we'll move from launching cloud servers using the > one-off libcloud-based python script I wrote to using the jenkins > jclouds plugin. (if for no other reason that to make sure that all of > the moving parts of this of any complexity are sensibly checked in > and have lifecycles that people can hack on.) > > ** If you like hacking on Java - this is a project that's both > helpful for us as consumers in OpenStack - but also is something > that effectively will allow other people to use OpenStack to manage > Jenkins build slaves... SO - come and hack on/improviding the > jenkins jclouds-plugin. It's at > https://github.com/jenkinsci/jclouds-plugin ** > > And then we need to apply this to swift/glance as well. > > > That's just testing API in a VM though, and doesn't get us to > testing actual bare-metal deployment or integration testing. At > Rackspace, we have some machines set aside at the moment, and have > had others offer chunks of machines to test various combinations of > things. At its heart, the abstract version of this looks fairly > identical to the smoketests job - pxe boot machines, shove version to > be tested on them, run tests. However, there are several moving bits > on the best way to actually do the how. At the moment, the fine folks > at rPath have a Jenkins installing and testing rPath OpenStack > images, so Mihai and I are going to look at getting that setup ported > to our Jenkins. However, although that will be an excellent test of > code, as our main target platform is Ubuntu, we're also looking at > doing a straight-up cobbler install using generated .debs. In any > case, this is the bit which is still in the planning and discussion > phase, but so far all of the conversations I've had with folks have > been great - and I'd love to get more folks involved in that (thus > this email) > > However- latent goal here is that whatever mechanism we're having > Jenkins use to deploy OpenStack onto real hardware should be > consumable and one that actual people might actually use - otherwise > what the heck are we testing? > > Additionally, as you may have surmised, it is also a goal to run as > much of this as possible from the OpenStack Jenkins, because that way > we can as a project choose to incorporate as much of the > feedback/results of various forms of testing directly in to branch > testing/approval as we want. For some things (spinning up 20 node > OpenStack clusters) doing it on every merge proposal or giving all > devs the ability to click a button and have it run on their branch > will likely be overkill - but if it turns out not to be, it would be > great to have the ability to do it. > > End goal is to have: - publicly accessible and usable system for > testing and build automation - resources that it uses to spin up > clouds in order to test them are themselves usable by people to spin > up clouds - tooling around this is done in a manner that makes us of > and contributes back to existing projects (jenkins plugins, patches > back to cobbler/orchestra/whatever) > > If you didn't read my _other_ long email from a few moments ago, > actual discussion of getting this done - and figuring out other > people's needs/tools and how to integrate them - is hopefully > happening next week right before the regular openstack-meeting. In > the mean time, please either flame on right here in list, or ping me > back personally. > > Thanks everyone! Monty > > > _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
