I'd like to open source them. They are dependent on some other internal tools, so I they wouldn't work out of the box, but I'll see if I can rip them apart and then get corporate sign off on open sourcing them.
On Mon, Aug 5, 2013 at 4:02 PM, Joshua Harlow <harlo...@yahoo-inc.com>wrote: > This does sound like a neat approach, a hybrid if u will. Might be > something to try :-) > > Sometimes I wish package managers were better, especially with regard to > 'complex' (not really that complex in reality) dependencies. > > Any possibility of opening up said makefiles?? Be interesting to look at > :-) > > From: Jay Buffington <m...@jaybuff.com> > Reply-To: OpenStack Development Mailing List < > openstack-dev@lists.openstack.org> > Date: Monday, August 5, 2013 3:37 PM > To: OpenStack Development Mailing List <openstack-dev@lists.openstack.org> > Subject: Re: [openstack-dev] [DevStack] Python dependencies: PyPI vs > distro packages > > I've been doing continuous deployment via rpms on RHEL 6u3 of glance, > keystone, > neutron and nova for about six months now. I used Anvil for the first > three > months, but it required constant updating of dependency versions and it > didn't > support quantum. Also, yum is terrible at managing dependency ranges, > which > openstack components often have (e.g. SQLAlchemy>=0.7.8,<=0.7.9). I'm not > convinced DNF[1] will be much better. > > I found it easier to just build a virtualenv and then put all those > files in an > rpm. Then make non-python dependencies (like libxslt) dependencies of > that rpm. > Since we did that the number of daily packaging headaches have dropped > significantly. We have a bunch of Makefiles and jenkins jobs that drive > this > on every commit to components we care about. > > I was unaware of RDO (or it hadn't been released) when I switched our > builds > over. I'd take a serious look at it, but right now I'm happy with our > solution. > > [1] http://fedoraproject.org/wiki/Features/DNF > > > On Mon, Aug 5, 2013 at 10:03 AM, Dean Troyer <dtro...@gmail.com> wrote: > >> [Moving a discussion from https://review.openstack.org/40019 to the ML >> to get a wider audience] >> >> We've been around this block more than once so let's get it all >> documented in one place and see where to go next. Skip down to >> ############# for more actual discussion... >> >> Given: >> * OpenStack now has an official list of Python package versions >> required in https://review.openstack.org/p/openstack/requirements >> * This list is rarely completely available in any packaged Linux distro >> * Developers like new packages that fix their immediate problem >> * Packagers dislike the treadmill of constantly upgrading packages for >> many reasons (stability, effort, etc) >> * Running OpenStack on certain long-term-stability distros (RHEL6) is >> seriously a challenge due to the number of out-of-date components, >> specifically here many of the python-* packages. >> * Fedora and RHEL6 have a nasty configuration of telling pip to >> install packages into the same location as RPM-installed packages >> setting up hard-to-diagnose problems and irritating sysadmins >> everywhere. FTR, Debian/Ubuntu configure pip to install into >> /usr/local and put '/usr/local/lib/python2.7/dist-packages' ahead of >> '/usr/lib/python2.7/dist-packages' in sys.path. >> * Replacing setuptools on RHEL6 illustrated another issue: removing >> python-setuptools took with it a number of other packages that >> depended on it. >> * OpenStack devs are not in the packaging business. This has been >> decided [citation required?]. Fortunately those in the packaging >> business do contribute to OpenStack (THANKS!) and do make a huge >> effort to keep up with things like the Ububntu cloud Archive and Red >> Hat's RDO. >> >> The last week or so of attempting to install Python prereqs from >> requirements.txt and installing pip 1.4 to support that rather than >> re-invent the wheel and all of the fallout from that illustrates >> clearly that neither approach is going to solve our problem. >> >> Summary of the discussion in the review (paraphrased, read the review >> to see where I got it wrong): >> >> * packages are evil: we develop and gate based on what is available in >> requirements.txt and a non-zero number of those are only in PyPI >> * Anvil solved this already: resolves requirements into the RPM >> package list and packages anything required from PyPI >> * ongoing worries about pip and apt/rpm overwriting each other as >> additional things are installed >> * packages are necessary: >> >> ############# >> >> My specific responses: >> >> * proposals to use a tool to automatically decide between package and >> PyPI (harlowja, sdague): this works well on the surface, but anything >> that does not take in to account the dependencies in these packages >> going BOTH ways is going to fail. For example: on RHEL6 setuptools is >> 0.6.10, we want 0.9.8 (the merged release w/distribute). Removing >> python-setuptools will also remove python-nose, numpy and other >> packages depending on what is installed. So fine, those can be >> re-installed with pip. But a) we don't want to rebuild numpy (just >> bear with me here), and b) the packaged python-nose 0.10.4 meets the >> version requirement in requirements.txt so the package will be >> re-installed, bringing with it python-setuptools 0.6.10 overwriting >> the pip installation of 0.9.8. >> * separate proposal to build meta-packages (iwienand, >> https://review.openstack.org/39862): this would be an effective >> work-around for the problems on Fedora and RHEL6 and protect against >> the re-install problem, but this is solving a distro packaging problem >> in an upstream testing tool. We would also need to manage these >> individually and not in a single large package, or at least split >> python-crypto and python-lxml out of the proposed package as they may >> not be installed in all configurations. >> * about-face on all-packages to all-PyPI (mtaylor): I'm still on the >> use packages where possible side but DevStack specifically is not in >> the packaging business. If it were we'd do what Java folk (*sorry*) >> have long taken the approach for non-trivial apps to just include the >> whole damn runtime. Write once duplicate everywhere. >> >> I want to propose moving forward with the following guidelines: >> * Keep a list of specific distro packages that we want to use in favor >> of PyPI for specific reasons like not recompiling native modules, i.e. >> python-crypto except on RHEL, etc >> * Continue to factor the prereq setup out of stack.sh such that >> requirements.txt is satisfied one way or another before it begins to >> install OpenStack. tools/install_prereqs.sh and tools/install_pip.sh >> are the prototypes for this. Each distro gets a chance to get it right >> here whether by package or by PyPI. >> >> tl;dr: no easy solution >> >> dt >> >> -- >> >> Dean Troyer >> dtro...@gmail.com >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
