Hi all! First of all, I spoke about something yesterday in terms of being a done-deal, when it should have been prefaced with "I'm looking at". My bad, and I certainly apologize for freaking anyone out.
That being said - there are several issues with the current packaging setup that I've been tasked with tackling. This email will outline the issues, and then I'll discuss what I've been thinking in terms of solutions. ***** This is a proposal. We want to get feedback on this before the summit but will not be changing the way we do packaging before the summit. ***** For reference, the current setup is: Jenkins creates source packages from a team (~openstack-ubuntu-packagers) packaging branch for every trunk commit for Ubuntu and uploads them to a PPA on Launchpad Let me start be enumerating the issues with the current setup that we're trying to solve. (These are not in order of importance) - Lack of Debian support. One of the largest current deployments of OpenStack is Rackspace, and they deploy on Debian. (disclaimer, they also pay my salary) This means they are internally maintaining their own set of packages, and not interacting with ours at all. That sort of sucks. - Lack of collaboration. Related to the above, but slightly different. Since the Rackspace internal deployment team have to maintain their own packages, it means they are not using the published packages and we are not benefiting from their real-world deployment experience. Additionally, the Cloudbuilders consulting team doesn't use our published packages, and similarly we aren't really benefiting from them directly working with us on packaging. - PPAs are Async. This makes integration testing harder. We're about to roll out deployment/integration testing, which involves installing the latest set of packages in environments and then testing that it works. In that workflow, the queue/async nature of using LP as our package builder becomes more problematic. I cannot trigger a direct test of a set of packages as part of the testing of a commit if part of the process is "upload into the LP queue and then wait for a while" - especially since LP still has not added any sort of callback mechanism to report completion. This means Jenkins is locally building packages anyway so that it can pass them to deployment environments to test them. If it's building them locally, and that's what's being tested, why wouldn't that be what's published? - Divergent dev process from the rest of OpenStack. The PPB just recently voted that all OpenStack projects must use the same set of tooling. Now granted, packaging is not a core project, so we could get legalistic and claim that we're not required to comply - but that seems a little disingenuous - especially since I was a strong proponent of the one-set-of-tooling model. Additionally, since we have a strong code review/testing system that's integrated in to our build and deployment system, it seems very odd not to take advantage of it. (having new packages created triggered by both approval of packaging branch changes AND new upstream tarballs seems kindof ideal, no?) - Not integrated with CI. PPA build errors are not reported inside of our CI system where all of the rest of our errors are reported. Additionally, those build errors are not reproducible by devs locally, as they cannot spin up a launchpad PPA builder. - Lack of RPM support. Turns out RPM distros are still kind of popular, and at least one of our major contributors (hi, NTT) rolls out on RHEL/CentOS, and one of our partners who is going to be contributing testing hardware (MSFT/Novell) is associated with a distro (SuSE) that is RPM based. We're a few steps away from this, but it warrants mentioning. - Murky backported deps. Some of the backported deps in the PPAs came from a dir on my laptop. Some are from Theirry's laptop. Some are from Soren's laptop. There is neither documentation on how they got there, how we decided to put them there, how we built them or how to continue to collaborate on them. - Direct injection to Ubuntu not via Debian. The publically stated best-practice for things entering Ubuntu is to be added to/uploaded to Debian and then synced, unless there is some compelling reason not to do this. In this case, we had Ubuntu devs and no Debian devs in the core, so we keep improving packages and adding packages to Ubuntu but are not doing the same to Debian. - PPA Fragmentation. The PPB voted that we are one project, yet we have a separate PPA for each project. Ok. I lied - those might have been in a general order of importance. Turns out though, that in terms of addressing them, they fall in to two different groupings: 1) How we manage and work on the packaging code itself 2) How we build and publish packages The two are related, but not linked. Of the two, the most absolutely important is 2, so I'll address it first. I'm also going to be a bit more blunt about it than the other one. == How we build and publish packages == This deals with: - Lack of Debian support. - PPAs are Async. - Not integrated with CI. - Lack of RPM support. - PPA Fragmentation. I'm going to be rolling out http://packages.openstack.org, which will at first contain packages for lucid, maverick, natty, oneiric, squeeze and wheezy. There will be at the very least a trunk repo and a repo per-release (so a diablo one) Will probably use the debian trick of pointing trunk at diablo at first, and then creating an essex repo and pointing essex at that - but I'm up for suggestions there. We have a Jenkins, we have plenty of builder slaves and we have pbuilder for chroot clean builds - there is no compelling reason for us to wait on the Launchpad build queue. AND - I can hand a puppet module to a dev that they can run on a cloud server and they can have access to an env which is EXACTLY like the one that build errors that they experience occured in. So big wins here are: - single place to get OpenStack as a cohesive whole - addition of pure Debian packages - possibility of adding YUM repos to packages.openstack.org - synchronous publishing for better testing - end to end visibility of what build actions were taken Note that there is nothing stopping PPA packages from existing - we could even have jenkins continue upload things to PPAs as well - it's just not really a primary target. == How we manage and work on the packaging code itself == Now to the one that's a little more up in the air because it's the one that's going to cause people to throw their hands up in the air and scream and gnash their teeth. Sigh. Sorry. To address: - Lack of collaboration. - Divergent dev process from the rest of OpenStack. - Murky backported deps. - Direct injection to Ubuntu not via Debian. I'm suggesting: - we move all packaging to git - we manage it consistently via git-buildpackage - we code review packaging branches in gerrit - we publish packages only through reviewed branches Because of the last, point, I'd like to point out that this applies to backported packages we need in our repo as well. Building any package from the set of important OpenStack pacakges should go through the same process. Managing the packaging for our packages and for backports will be slightly different. What exactly does that mean? For our packages: As Thomas Goirand suggested, we will add a packaging branch to the repo of each of our projects and have git-buildpackage do the right thing. We can have gerrit set different ACLs for the packaging branch so that a different set of folks (the packaging team) can handle the reviewing of that branch. Combining this with a pristine-tar and having Jenkins push some tags back to the repo on package release, it means that we can trace any release of software back to our code repo, and that only the repo is needed to re-create any particular release. The jenkins job for this is triggered off of both new commits to trunk and to new commits to packaging. For backported packages: Having branches on top of upstream source is almost useless here, as in most cases we're doing one-off backports. So here we simply make a new repo via git-import-dsc, upload it and register it with jenkins. In the common case, there won't be much more work to do - but in the ongoing case, it means that anyone in our dev community knows where to go to get at packaging of specific versions of things carried in our repo. The jenkins jobs for these are triggered off of new commits to packaging. For both: When Jenkins is triggered, it will pull the branches and build packages for all available targets in base chroots using git-pbuilder/cowbuilder. Each build will have its full log in Jenkins just as any other build. >From a repo perspective, for each target, Jenkins will attempt to build using packaging in the debian branch unless there is a branch named debian/${target} - so if the packaging needs to be overridden for, say, lucid (because it's older than dirt) It's easy enough to just create a debian/lucid branch and leave debian to handle the other branches. These packages are then all published to packages.openstack.org and jobs testing deployment are kicked. At a point in the future (if we start doing pre-merge deployment testing) we'll need to make a pending archive so that we can re-publish same versions due to failure -but we're not there yet. As a bonus, git-buildpackage does some nice tagging of things, which we can have Jenkins push back, so we can keep our repos up to date with meta information. Since these are on ancillary branches, that pushing doesn't have to interrupt normal dev work. Incidentally, since that meta info is in the repo, we cease to need the tarball_versions and ppa_versions files that we currently manage on the filesystem in jenkins. I know there are other ways to accomplish some of these things, but this is the one I'm proposing. The fact that it fits in to our development methodology is appealing, as is the possibility of getting pvo and jesse and thomas looped in to working directly on this set of packages. Monty _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
