Hi Clint, Thanks for your contribution to this thread.
On 06/04/2015 10:35 PM, Clint Adams wrote: > On Wed, Jun 03, 2015 at 04:30:17PM -0400, Sean Dague wrote: >> The closer we can get logic about what a service should look like on >> disk back into that service itself, the less work duplicated by any of >> the installers, and the more common OpenStack envs would be. The fact >> that every installer / package needs to copy in a bunch of etc files >> (because the python packages don't do it) has always seemed rather odd >> to me. > > I agree with this, and given the disparate and seemingly contradictory > goals driving these discussions, I think it will be exceedingly > difficult to make everyone happy. I don't think anyone involved in the packaging of OpenStack has expressed disparity or contradiction. Quite the opposite: it is my strong opinion that all parties involved are on the same page. > So here's my suggestion: > > 1. Maintain all important data for packaging as first-class members of > the respective repositories. Initscripts, systemd service files, > licensing (SPDX?), and so on should be in the master branch of each > project. Are you here saying we should move startup scripts upstream, and not on the packaging repos? If so, that's a bad idea. Let me explain. The init scripts used to be hard to maintain because they were many, but since Debian & Ubuntu are using automatic generation out of a tiny template (with sysv-rc, systemd and upstart all supported), this is a problem solved. If individual projects are getting into the business of publishing their own startup scripts, I doubt that we would even use them, because having a central place to patch all of the startup scripts at once (ie: openstack-pkg-tools) is much better than having to maintain each and every startup script by hand. As for the licensing, I agree here. I have multiple times express my opinion about it: the project as a hole needs to make some progress, as it's nearly impossible for downstream distributions to second guess who is the copyright holders (please pay attention: copyright holders and licensing are 2 separate things...). Though SPDX?!? Do you know the famous quote (Jay Pipes) "Get your dirty XML out of my Json" ? :) And there's also the fact that Debian uses a different parseable format. Not sure what the RPM folks use, but I suppose that's embedded in a .spec file. Also, all of OpenStack is mostly using the Apache-2.0 license, the licensing issues are with (build-)dependencies, and that, we can't solve it. > Just enough glue should be present such that functional > packaging can be programmatically generated from HEAD with debdry > or similar tooling, and this is how test jobs should operate (f.ex.: run > debdry, mangle version number to something unique, build package in > chroot or equivalent, store output for use in other testing). I already use pkgos-debpypi (from openstack-pkg-tools) to automate a big chunk of the Python module packaging. Though the automated thing can only drive you so far. It wont fix missing dependencies in requirement.txt, wrong lower bounds, SSLv3 removal related issues, and all of this kind of issues which we constantly need to address to make sure all unit tests are passing. The thing is, unit & functional tests in OpenStack are designed for Devstack. I supposed you already saw the "It worked on Devstack" XKCD t-shirt... Well, I'm sure all package maintainers of OpenStack have this in mind every day! :) As you are a DD, you know it too: getting a correct debian/copyright also represent some work, which cannot be automated, or the FTP masters will really hate you. :P Plus there's also the fact that sometimes, distributions don't use the matching versions. For example, Debian Jessie was released with Django 1.7 and OpenStack Icehouse, and I had to work on patching Horizon to make it work with it (at the time, Horizon didn't work with something higher than 1.6). The same thing happened with SQLAlchemy. This has slowed down a little bit over the years, but we also used to spend most of our time simply packaging new dependencies. We already have a big amount of redundancy (alembic vs sqlalchemy-migrate, WSGI frameworks, nose vs testr, pymysql vs mysqlclient vs mysql-python, you name it...). And each new project, with its specificities, bring new dependencies. I used to say that 80% of the packaging work is spent there: packaging new stuff. These days, it has a bit shifted to packaging oslo and client libs, which is a good thing (as they are respecting a standard, so we have less surprise). Though what actually represent an OpenStack release has grown bigger. After Jessie was released, uploading all of Kilo to Sid took me about 3 or 4 days, and maybe about the same amount of time to upload it to the official Jessie backports (all done in dependency order, with as few dependency breakage as possible...). I really hope that the effort on having a gate on the lower bounds of our global-requirements.txt will help here. But it wont solve all issues. So, that's where all the time is spent. Yeah, we can do some level of automation. But believing we can do *ALL* with automation is often a naive thought of people who never tried. I welcome you to join the packaging effort and enhance the toolset we have so we get closer to that goal though! :) > This way OpenStack could > > - produce and consume its own package artifacts for testing changes of > varying complexity > - get early warning of changes which will break packaging, insanity in > dependency graphs, and so on That's some of the goals of this proposal, yes. > Distributors could > > - clone the upstream repos and branch to add any special tweaks, then > potentially run the same automation to get source/binary packages > - collaborate upstream to fix things of common utility At least for Debian & Ubuntu, we hope to push absolutely all of our packaging upstream, and only rebuild and publish the final result for the stable branches (well, I can only speak for myself, so I'll say that this is what I want to do in Debian...). Even if we can do this for all but the core projects, then that's a huge win (since it represents the biggest chunk of the work). I'll do as much as possible so that Mirantis OpenStack also follows upstream packaging branches too, and also contributes upstream. Cheers, Thomas Goirand (zigo) P.S: Will you get involved in the packaging for Debian as well? __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
