Hey everybody, Here's a quick write up on what we have at the moment and what the plan is moving forward, as it stands today. In case you weren't aware, the Jenkins instance sits at:
http://jenkins.openstack.org If you think you want to help administer/hack on our Jenkins, that is managed through the openstack-ci-admins team on Launchpad. As most of you are aware, the main trunk branches for nova, swift, glance and burrow are all managed by Jenkins and Tarmac. What this means in short is that jenkins/tarmac finds approved merge proposals, runs the in-tree unit tests on them, and if successful, pushes back to trunk. Additionally, we have jenkins jobs which run pep8, pylint and code coverage jobs and produce reports. For instance: http://jenkins.openstack.org/job/nova-coverage/815/cobertura/? Once the code has been merged, we have jenkins jobs which produce tarballs, and debian source packages which are pushed up to launchpad PPAs so that effectively every trunk commit winds up having an associated package. There is also a feature that it seems no one knew about which allows developers to submit branch URLs to jenkins to have it run its tests on that branch. For each project, this is ${project_name}-param, so for instance: http://jenkins.openstack.org/job/nova-param/ Will allow anyone in the ~nova team to submit a branch and have jenkins pull it and run on it what it would do via tarmac - helpful when fixing something tarmac complained about. We are planning to get rid of tarmac and get that integrated in as a Jenkins plugin, because otherwise supporting things like NTT's wish for stricter coverage metrics as branch gatekeepers, or smoketesting is preventing branches from getting in become really baroque to support. ** If you like hacking on Java - this should be a fun little project. Send me a note and I can outline what I was going to do and if you want it it's all yours. ** Moving forward, the big ticket item is testing not only in-tree unit tests, but installing things and testing that. Currently, we have jenkins spinning up a cloud server, copying the most recently built debs from the last successful build to that server, installing them, starting things up and then running the smoketests against the installed code. Currently the install/setup is being done by hand rather than via chef purely so that I could walk through what's actually needed to get a single-node minimal install working and collect information on bugs/workarounds that have to happen. http://jenkins.openstack.org/job/nova-smoketests The test failures here are due to config issues which should go away once I migrate that job to using chef for the node code - so I expect that to go green soon. The next step then is to replace the use of shell commands over ssh with the chef recipes. (now that I have a good handle on what's going on/needed there, following the existing chef automation work is much more fruitful) After that, we'll move from launching cloud servers using the one-off libcloud-based python script I wrote to using the jenkins jclouds plugin. (if for no other reason that to make sure that all of the moving parts of this of any complexity are sensibly checked in and have lifecycles that people can hack on.) ** If you like hacking on Java - this is a project that's both helpful for us as consumers in OpenStack - but also is something that effectively will allow other people to use OpenStack to manage Jenkins build slaves... SO - come and hack on/improviding the jenkins jclouds-plugin. It's at https://github.com/jenkinsci/jclouds-plugin ** And then we need to apply this to swift/glance as well. That's just testing API in a VM though, and doesn't get us to testing actual bare-metal deployment or integration testing. At Rackspace, we have some machines set aside at the moment, and have had others offer chunks of machines to test various combinations of things. At its heart, the abstract version of this looks fairly identical to the smoketests job - pxe boot machines, shove version to be tested on them, run tests. However, there are several moving bits on the best way to actually do the how. At the moment, the fine folks at rPath have a Jenkins installing and testing rPath OpenStack images, so Mihai and I are going to look at getting that setup ported to our Jenkins. However, although that will be an excellent test of code, as our main target platform is Ubuntu, we're also looking at doing a straight-up cobbler install using generated .debs. In any case, this is the bit which is still in the planning and discussion phase, but so far all of the conversations I've had with folks have been great - and I'd love to get more folks involved in that (thus this email) However- latent goal here is that whatever mechanism we're having Jenkins use to deploy OpenStack onto real hardware should be consumable and one that actual people might actually use - otherwise what the heck are we testing? Additionally, as you may have surmised, it is also a goal to run as much of this as possible from the OpenStack Jenkins, because that way we can as a project choose to incorporate as much of the feedback/results of various forms of testing directly in to branch testing/approval as we want. For some things (spinning up 20 node OpenStack clusters) doing it on every merge proposal or giving all devs the ability to click a button and have it run on their branch will likely be overkill - but if it turns out not to be, it would be great to have the ability to do it. End goal is to have: - publicly accessible and usable system for testing and build automation - resources that it uses to spin up clouds in order to test them are themselves usable by people to spin up clouds - tooling around this is done in a manner that makes us of and contributes back to existing projects (jenkins plugins, patches back to cobbler/orchestra/whatever) If you didn't read my _other_ long email from a few moments ago, actual discussion of getting this done - and figuring out other people's needs/tools and how to integrate them - is hopefully happening next week right before the regular openstack-meeting. In the mean time, please either flame on right here in list, or ping me back personally. Thanks everyone! Monty _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
