On 08/24/2011 01:11 PM, Soren Hansen wrote: > 2011/8/24 Monty Taylor <mord...@inaugust.com>: >> - Lack of Debian support. One of the largest current deployments of >> OpenStack is Rackspace, and they deploy on Debian. (disclaimer, they >> also pay my salary) This means they are internally maintaining their own >> set of packages, and not interacting with ours at all. That sort of sucks.. > > This is false. The packages are perfectly capable of building and > running on Debian (barring currently unknown bugs). Whether *the rest > of Debian* is ready for it is a different story. We chose Ubuntu to > begin with not just because we have Ubuntu core devs on the team, but > because Ubuntu was the only distro that met our needs: > a) A release cycle that lined up with what we wanted. > b) A fresh set of packages in its current release. > c) A sane, dependable way to get required changes into development releases. > > Debian releases are rare and unpredictable. If you want a fresh set of > libraries on top of which you can build your stuff any Debian release > is out of the question and I don't think having a moving target (like > unstable or testing) as a deployment target is a very sound idea. > Imagine Cactus had been targeted for testing and we tested it and it > all worked on release day, but the next day someone changed something > in testing, but Cactus was already out of the door, frozen. That'd > suck. > > Alternatively, you have to maintain a set of backports, but that's a > major pain (and it only gets worse over time).
We already maintain a set of backports. This will not change. Also, I think that our definitions of some things are different here. I'm not talking about reference platform for inclusion upstream. I'm not talking about "I'm on Ubuntu Oneiric and I want to install OpenStack, so I'm going to apt-get install openstack from the main Ubuntu repository" I imagine it will be literally YEARS before anyone does that. I expect that people running OpenStack will, as step one, add the OpenStack repo, or the OpenStack PPA, or wherever we publish our latest and greatest packages. We released Cactus 6 months ago, and we did it in such a way as to get it included into Ubuntu - and less than a month later the general consensus is "cactus is unusable" So I'm talking about repository lifecycle from an upstream perspective with the assumption that I will be maintaining a repository from which my consumers can run OpenStack. Futher, I am assuming that some of them will want to run current OpenStack on Lucid. (hi Rackspace Cloud Files) or on Debian (hi upcoming Rackspace Cloud Servers) or RHEL6. As upstreams, it is incumbent on us to provide a complete experience for that. This, btw, is the disconnect in intentions that comes up at every UDS when I put on my hat as upstream (formerly from Drizzle, before that from MySQL, now from OpenStack) - which is that main inclusion is all well and good, but I don't actually care. In fact, the only reason I really care about main inclusion for OpenStack for Ubuntu is because Canonical has a (asinine) requirement that only software in main can run in their data centers, and I would like for Canonical devs to be able to run internal services using OpenStack rather than Eucalyptus. Other than that, I do not think any of the consumers of OpenStack will use the packages directly from Ubuntu. (especially since to do so in the next few months would mean deploying Oneiric server, which I think even fewer people will do. Data centers tend to like to deploy servers running latest RHEL or latest Ubuntu LTS and then deploy up-to-date packages for the services that that particular machine runs. All of that to say: - PPAs do not allow me to upload packages to be built for wheezy or squeeze nor host them. - We need to provide a set of packages that someone who is basing their environment on squeeze can be assured will work. - We will be maintaining backported dependency packages from now until the cows come home. >> - Lack of collaboration. Related to the above, but slightly different. >> Since the Rackspace internal deployment team have to maintain their own >> packages, it means they are not using the published packages and we are >> not benefiting from their real-world deployment experience. >> Additionally, the Cloudbuilders consulting team doesn't use our >> published packages, and similarly we aren't really benefiting from them >> directly working with us on packaging. > > Rackspace isn't doing their own packaging because of (lack of) Debian > support. If they did, they'd have realised almost immediately that the > packages actually build on Debian. They're doing it because there'll > supposedly be differences in the behaviour Ubuntu (or any other distro > for that matter) will want and what Rackspace will want from the > packages. I've becried this divergence countless times, but to no > avail. AFAIK, the packages do _not_ build on debian, nor have they for a while. Actually, this is exactly the point - we are not building them on debian, so we do not notice when they break. That's part of the point of a CI system - if it isn't an output target, no one will realize when they break it. > That cloudbuilders aren't using the packages either.. I don't really > know what to say about that. > >> - PPAs are Async. This makes integration testing harder. > > PPAs were chosen because > a) they are almost exactly identical to the builders for Ubuntu > proper, so the on-commit-to-trunk builds serve as way to know > immediately if we introduce a change that will break when uploaded to > Ubuntu proper. So, they're the closest we'll get to integration tests > for real Ubuntu builds, > b) we don't have to maintain them, and This was a concern when they were chosen. Now? Not a big deal. We maintain lots of things at this point. > c) adding a PPA as a source to an Ubuntu system is incredibly > straight-forward and a very well-supported operation. So is adding another repo. add-apt-repository "deb http://packages.openstack.org/trunk main" works just fine. Actually, the fact that add-apt-repository only does the key fetching for PPAs and not for other repos is sort of ridiculous and in-bred, but I've bitched at canonical enough for one day (don't get me started on the broken 2.6.6-3~ thing) >> If it's building them locally, and that's what's being tested, why >> wouldn't that be what's published? > > Because of c) above and because PPAs are trustworthy. There's no way > you can sneak extra packages into the chroot or otherwise compromise > its integrity, there's no way you can reach the internet from the > builders (and vice versa), and they're built in a trusted environment > (run by people who are used to managing this exact sort of > infrastructure). This is not unique to PPAs. Our repo can be just as trustworthy. In fact, we're shipping software which runs large portions of your data center, so if you don't trust us to make it, well, you're sort of screwed. Same as our current unit-test runners ... the system is locked down, the dev can't sneak stuff in. BTW - the "can't reach the internet from the builders" is only really a concern in a project like Debian and Ubuntu where you have random people uploading random un-reviewed things. In our system, where things _ONLY_ hit package builders as a result of peer review, we should actually be in better shape. (because we don't have to solve the general case like they do) >> - Divergent dev process from the rest of OpenStack. The PPB just >> recently voted that all OpenStack projects must use the same set of >> tooling. Now granted, packaging is not a core project, so we could get >> legalistic and claim that we're not required to comply - but that seems >> a little disingenuous - especially since I was a strong proponent of the >> one-set-of-tooling model. Additionally, since we have a strong code >> review/testing system that's integrated in to our build and deployment >> system, it seems very odd not to take advantage of it. (having new >> packages created triggered by both approval of packaging branch changes >> AND new upstream tarballs seems kindof ideal, no?) > > The packaging effort is far, far from an OpenStack-only operation. > It's a shared effort with the Ubuntu team. We chose tools and > processes for packaging that lined up with what is customary in Ubuntu > in an effort to build confidence in the tools and automation we built. > It maps very, very well with how things are usually done in Ubuntu. It does map very very well with how thing are usually done in Ubuntu. It does _NOT_ map very very well with how things are (coming to be) done in OpenStack. As I mentioned earlier (and this comes from someone with a great deal of love for Ubuntu) Ubuntu's goal and needs are not necessarily ours. Ubuntu devs are primarily focused on release+1 (because that's their job) WE are primarily focused on release, release-1 and last-LTS. This makes things like 2.6.6-3~ thing have very different emphasis and meaning depending on which side of the fence you stand. So my FIRST concern is making sure that the software that we produce meets the needs of our community, our devs and our users. My second concern is making things easier for Ubuntu. If the Ubuntu devs decide that they cannot work with OpenStack tooling, that's fine - although I will be very sorry to see them go. I'm more concerned with getting packages produced from us that match the needs of people actually deploying and using OpenStack on a regular basis. Right now, that means notmyname, pvo, anotherjesse, jmckenty >> - Not integrated with CI. PPA build errors are not reported inside of >> our CI system where all of the rest of our errors are reported. >> Additionally, those build errors are not reproducible by devs locally, >> as they cannot spin up a launchpad PPA builder. > > I don't understand. > a) Building a chroot and setting up sbuild is a matter of installing > one pacakge and running one command, Still doesn't get you exact same as a PPA builder. The PPA buider has networking turned off, which confused the sphinx build a while back until we figured out what was going on. The PPA build also has not only the packages inside of itself, but also other build-dep repos it can use. Making sure you get a chroot with the right repos takes a decent amount of knowledge about launchpad and ppas, actually. > b) further up you argued that building packages ourselves is pretty > much identical to building in a PPA and further down you say you have > puppet magic to create a build environment for devs. Those two things > put together sure sound to me like they can get infinitesimally close > to the same sort of environment as you see in Launchpad, Yes. Same sort of env. > c) it's going to be exactly as difficult/easy to create a build > environment that mimicks what you're doing in Jenkins, isn't it? Nope, not really. That's pretty darned simple. (right now it's like, three commands, I can definitely lower the bar there) Also, in jenkins I can set up parameterized builders which devs can submit code branches to and have them build in the actual jenkins env. That is MUCH harder to accomplish in Launchpad. More importantly, though, is the fact that all of the build output in the Launchpad PPA is hidden. Didja know that Glance hasn't built for Maverick or Lucid in a bit? That python-novaclient failed all of its latest PPA builds? No? Of course not - it's not in Jenkins, where all of the rest of the success/failure reporting is. For the same reason that I unreasonably yelled at Dan Prince at the last ODS for having built a parallel CI system that wasn't part of our CI system, the fact that we have important parts of our infrastructure building elsewhere with no reporting back into our centralized build system is becoming more and more singularly unacceptable. >> - Lack of RPM support. Turns out RPM distros are still kind of popular, >> and at least one of our major contributors (hi, NTT) rolls out on >> RHEL/CentOS, and one of our partners who is going to be contributing >> testing hardware (MSFT/Novell) is associated with a distro (SuSE) that >> is RPM based. We're a few steps away from this, but it warrants mentioning. > > This probably warrants a separate discussion. As pointed out further > up, Ubuntu was chosen in part because it's very up-to-date. With a > release every 6 months, we're never far behind in terms of access to > dependencies in the most recent release, and since our release cycles > line up, we can *ensure* availability of the dependencies we require > in the upcoming release. If > RHEL/CentOS/ScientificLinux/WhatEverOtherFlavourOfRedHatYouPrefer > becomes a supported target, will this block stuff from landing in Nova > until RHEL catches up with the rest of the world in terms of libraries > upon which we depend? Again - we're going to be maintaining repos and backports. RHEL is a good example, but the same applies to Ubuntu - the distros will NEVER keep up with our needs, because their needs are different. >> - Murky backported deps. Some of the backported deps in the PPAs came >> from a dir on my laptop. Some are from Theirry's laptop. Some are from >> Soren's laptop. There is neither documentation on how they got there, >> how we decided to put them there, how we built them or how to continue >> to collaborate on them. > > Sorry, what? They're all built in PPAs. That's about as clean-room as > it gets. Everyone is free to go and see and inspect them. The diff > between that version upon which the backport is based and what got > uploaded is trivially easy to extract. Documenting how it's done > sounds pointless. Anyone who's familiar with Ubuntu/Debian packaging > will find no surprises. If people want to look at OpenStack code, it's > not our job to document what Python is all about. If people want to > see how I backported something, it's not my job to teach them > packaging first. nono. Not how they're built - where the source tree is that you (or I) built the source package from that is built in the builders that aren't controlled by the Jenkins that controls everything else we do. If you (or I) do a one-off backport, we then upload a source package and then, often, don't shove the code anywhere. That's because a lot of the time it's easiest just to do dget, hack version, debuild -S, dput. We have no way of knowing if there is intention of further work, or where the canonical location of that work is. If I wanted to do an additional change to python-kombu, should I just grab the dsc from our PPA and upload a new one? Or did you already import that in to a bzr repo? Who knows? How can they find out? (and yeah, UDD was supposed to solve this - except it didn't) >> - Direct injection to Ubuntu not via Debian. The publically stated >> best-practice for things entering Ubuntu is to be added to/uploaded to >> Debian and then synced, unless there is some compelling reason not to do >> this. In this case, we had Ubuntu devs and no Debian devs in the core, >> so we keep improving packages and adding packages to Ubuntu but are not >> doing the same to Debian. > > For the umpteenth time, this is not about OpenStack. It's about the > dependencies. Getting OpenStack itself into Debian was never the hard > part. Getting all the dependencies into Debian was. Not just libraries > that weren't in Debian at all (like gflags, which I got uploaded to > Debian and synced into Ubuntu afterwards), but existing libraries in > Debian that we needed either a newer version of or a patch applied to. > In Ubuntu we have the power to upload these newer versions or patches > or whatever, by implicitly accepting responsibility for it if it > should happen to cause problems. We have no way to even begin to > ensure that we can do this in Debian. Debian packages have > maintainers. The maintainers decide whether to accept a patch or not. > If they're not comfortable with a patch, rejecting it is entirely > their prerogative (and for good reason! They're the ones who > ultimately have to support it). In every Ubuntu release since we > started work on OpenStack, I've added at least one patch to libvirt to > support something we needed for OpenStack. I've submitted them > upstream (to libvirt) as well, but we can't stop everything while > waiting for libvirt to make a release, the Debian maintainer to pick > it up, and for Ubuntu to sync it. Yes. I understand all of that ... but if we START with debian-oriented packages in our own repo and send patches upsteam (as you've already been doing) we get Debian support as a target. Those same patches can then also be applied to our packages for oneiric and maverick and natty and lucid, which can also be uploaded to our repo. And then, as we either have time or inclination or motivation from downstream, we can get libraries updated in both debian and ubuntu. Part of the problem here is that we have an institutional process which depends on a special quality of one of the OpenStack devs (namely, the ability of you or ttx to upload directly in to Ubuntu) That's FANTASTIC, but it can't really be the basis of ongoing OpenStack process, as it is not a process that someone else can just pick up. > Debian support is a massive undertaking and I'm thrilled to see it > happen, but I don't believe Debian is suitable as a reference platform > (if that means that failure there can block anything at all). I don't think you and I have the same view of the words "reference platform", but I believe I've already covered that. >> - PPA Fragmentation. The PPB voted that we are one project, yet we have >> a separate PPA for each project. > > For trunk builds, yes. Two reasons: > a) Avoid cascading failures: Nova depends on glance which depends on > Swift. It would be a shame if a bug in Swift had to bring everything > else to a screeching halt. But it's already bringing things to a halt in terms of actual deployments. Jesse just said to Jim and I yesterday as we were doing integration test setups "oh yeah, this stuff breaks all the time" The point of a good CI system is to head that off at the pass. Nothing about our current process should allow a bug in Swift to bring DEV to a screeching halt, but if it causes integration tests to bork, well, then people probably want to know about it, no? > b) If someone wants to help test the bleeding edge of Swift, but isn't > really interested in whatever breakage we introduce in Nova, having > separate PPA's makes it straightforward to just test one of them. I've heard this as a theoretical several times, but I haven't heard the actual call for "I REALLY want to install bleeding edge swift, with stable nova" on the same machine. In fact, the chances of running both swift and nova on the same box in a real environment are pretty low. Interface libraries, on the other hand, sure - except that new interface libraries should ALWAYS be able to talk to old versions of services. > For final releases, we put everything in the same PPA. I've written > tools to keep track of divergence between the PPA's and to bring them > in sync so the final consolidation doesn't hold any surprises. Putting > everything in the same PPA is a trivial change, though. Where are these tools? Where do they run? When? From what source do they bring PPAs in sync? How do they decide which version of eventlet needs to live in the nova and the swift PPA? From were do they get that version of eventlet? Monty _______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
