On Tue, Apr 19, 2016 at 12:24 PM, Ian Cordasco <sigmaviru...@gmail.com> wrote:
> > > -----Original Message----- > From: Thomas Goirand <z...@debian.org> > Reply: OpenStack Development Mailing List (not for usage questions) < > openstack-dev@lists.openstack.org> > Date: April 18, 2016 at 17:21:36 > To: OpenStack Development Mailing List (not for usage questions) < > openstack-dev@lists.openstack.org> > Subject: Re: [openstack-dev] [release][requirements][packaging][summit] > input needed on summit discussion about global requirements > > > Hi Doug, > > > > I very much welcome opening such a thread before the discussion at the > > summit, as often, sessions are too short. Taking the time to write > > things down first also helps having a more constructive discussion. > > > > Before I reply to each individual message below, let me attempt to reply > > to the big picture seen in your etherpad. I was tempted to insert > > comments on each lines of it, but I'm not sure how this would be > > received, and probably it's best to attempt to reply more globally. > > > > From what I understand, the biggesgt problems you're trying to solve is > > that managing the global-reqs is really time consuming from the release > > team point of view, and especially its propagation to individual > > projects. There's IMO many things that we could do to improve the > > situation, which would be acceptable from the package maintainers point > > of view. > > > > First of all, from what I could see in the etherpad, I see a lot of > > release work which I consider not useful for anyone: not for downstream > > distros, not upstream projects. Mostly, the propagation of the > > global-requirements.txt to each and every individual Python library or > > service *for OpenStack maintained libs* could be reviewed. Because 1/ > > distros will always package the highest version available in > > upper-constraints.txt, and 2/ it doesn't really reflect a reality. As > > you pointed out, project A may need a new feature from lib X, but > > project B wont care. I strongly believe that we should leave lower > > boundaries as a responsibility of individual projects. What important > > though, is to make sure that the highest version released does work, > > because that's what we will effectively package. > > > > What we can then later on do, at the distribution level, is artificially > > set the lower bounds of versions to whatever we've just uploaded for a > > given release of OpenStack. In fact, I've been doing this a lot already. > > For example, I uploaded Eventlet 0.17.4, and then 0.18.4. There was > > never anything in the between. Therefore, doing a dependency like: > > > > Depends: python-eventlet (>= 0.18.3) > > > > makes no sense, and I always pushed: > > > > Depends: python-eventlet (>= 0.18.4) > > > > as this reflects the reality of distros. > > > > If we generalize this concept, then I could push the minimum version of > > all oslo libs into every single package for a given version of OpenStack. > > > > What is a lot more annoying though, is for packages which I do not > > control directly, and which are used by many other non-OpenStack > > packages inside the distribution. For example, Django, SQLAlchemy or > > jQuery, to only name a few. > > > > I have absolutely no problem upping the lower bounds for all of > > OpenStack components aggressively. We don't have gate jobs for the lower > > bounds of our requirements. If we decide that it becomes the norm, I can > > generalize and push for doing this even more. For example, after pushing > > the update of an oslo lib B version X, I could push such requirements > > everywhere, which in fact, would be a good thing (as this would trigger > > rebuilds and recheck of all unit tests). Though, all of this would > > benefit from a lot of automation and checks. > > > > On your etherpad, you wrote: > > > > "During the lead-up to preparing the final releases, one of the tracking > > tasks we have is to ensure all projects have synced their global > > requirements updates. This is another area where we could reduce the > > burden on the release team." > > > > Well, don't bother, this doesn't reflect a reality anyway (ie: maybe > > service X can use an older version of oslo.utils), so that's not really > > helpful in any way. > > > > You also wrote: > > > > "Current ranges in global-requirements are large but most projects do > > not actively test the oldest supported version (or other versions in > > between) meaning that the requirement might result in broken packages." > > > > Yeah, that's truth, I've seen this and reported a few bugs (the last I > > have in memory is Neutron requiring SQLA >= 1.0.12). Though that's still > > very useful hints for package maintainers *for 3rd party libs* (as I > > wrote, it's less important for OpenStack components). We have a few > > breakage here and there, but they are hopefully fixes. > > > > Though having a single version that projects are allowed to test with is > > super important, so we can check everything can work together. IMO, > > that's the part which should absolutely not change. Dropping that is > > like opening a Pandora box. Relying on containers and/or venv will > > unfortunately, not work, from my stand point. > > > > The general rule for a distribution is that the highest version always > > win, otherwise, it's never maintainable (for security and bug fixes). It > > should be the case for *any program*, not even just any OpenStack > > components. There's never a case where it's ok to use something older, > > just because it feels like less work to do. This type of "laziness" > > leads to very dangerous outcomes, always. > > > > Though I don't see any issue with a project willing to keep backward > > compatibility with a lower version than what other project do. In fact, > > that's highly desirable to always try to remain compatible with lower > > versions as much as possible, *if* it doesn't bring too much burden > > (think: we still have loads of Python 2.6 stuff like discover, argparse > > and such that needs to be cleaned-up). > > > > For the projects who aren't following the release cycles, it's ok if > > they bump their lower bounds at the condition that they keep what's in > > the upper-constraints.txt, so that we can release them together with the > > rest of OpenStack. For example, it's ok to require the latest oslo.utils > > 3.8.0 for Ironic 5.1.0. > > > > I also see no reason to completely freeze versions to the point were it > > is impossible fix bugs in the versionnings. I've seen that multiple > > times. At the distro level, we do address the issue anyway, so why > > bother? The only thing that counts for us, is to be able to release > > everything together. A good example would be that Neutron needing SQLA > > >= 1.0.12: we refuse to fix it, but the issue is still there, and we > > have to fix it in distros. > > > > Last, before I reply to each individual message: the attempt from Robert > > to test on lower bounds is really something that should be pushed > > forward. I would help a lot to actually know what the reality is, rather > > than doing second-guess and artificially set lower bounds for versions > > which we think may probably work. > > > > On 04/18/2016 02:22 PM, Chris Dent wrote: > > > targeting an > > > all-in-one install seems only to serve the purposes of all-in-one rpm- > > > or deb-based installations. > > > > Though that's what most everyone consumes. Or are you proposing that we > > completely stop publishing OpenStack in distros? > > > > Remember that in distros, there's only a single version of a library at > > any given time, at the exception of transitions (yes, in Red Hat it's > > technically possible to install multiple versions, but the policy > > strongly advocates against this). > > > > Also, all-in-one is what I use in Debian in my CI, to make sure that > > everyone works together, with whatever is uploaded (see the > > openstack-deploy package in Debian). > > > > > Many (most?) people won't be doing those kinds of installations. > > > > Most users are consuming packages from distributions. Also, if you're > > using containers, probably you will also prefer using these packages to > > build your containers: that's the most easy, safe and fast way to build > > your containers. > > > > > If all-in-one installations are important to the rpm- and deb- based > > > distributions then _they_ should be resolving the dependency issues > > > local to their own infrastructure > > > > Who is "they"? Well, approximately 1 person full time for Debian, 1 for > > Ubuntu if you combine Corey and David (and Debian and Ubuntu > > dependencies are worked on together in Debian), so that's 2 full time > > for Ubuntu/Debian. Maybe 2 and a half for RDO if what Haikel told me is > > still accurate. > > > > So "we" wont handle it, even if "we" care, because we're already > > understaffed. > > > > > (or realizing that it is too painful and start > > > containerizing or otherwise as well). > > > > Distributions don't package containers, we offer packages that you can > > use to build them. The way you deploy is IMO orthogonal to packaging. > > > > > I think making these changes will help to improve and strengthen the > > > boundaries and contracts between services. If not technically then > > > at least socially, in the sense that the negotiations that people > > > make to get things to work are about what actually matters in their > > > services, not unwinding python dependencies and the like. > > > > Yes, it simplifies the life of developers. But it's going to be hell on > > earth for production if we discover a serious vulnerability in a > > component, which will be deployed N times, with N versions. Are we then > > going to provide N patches? The vulnerability management team is also > > understaffed... > > > > > One challenge I've run into the past is when devstack > > > plugin A has made an assumption about having access to a python > > > script provided by devstack plugin B, but it's not on $PATH or its > > > dependencies are not in the site-packages visible to the current > > > context. The solution here is to use full paths _into_ virtenvs. > > > > Please take a step back. Devstack and virtualenv are for development, > > not for production. That's not, and should not become, our target. > > Applying the reasoning you have there will *not* work for distributions. > > Hopefully, that's what Doug is proposing, as he would still like things > > to work for distros (right?). > > > > On 04/18/2016 03:24 PM, Hayes, Graham wrote: > > > It is also worth noting that according to the OpenStack User Survey > > > [0] 56% of deployments use "Unmodifed packages from the operating > > > system". > > > > It is my believe (and hope, as this would mean more users) that some > > OpenStack users don't even know about this survey, and just install > > what's in the distros. So 56% is the lowest estimate. Also, "Unmodifed > > packages" is the answer. Maybe some are deploying "slightly modified > > packages, but mostly what's upstream", which would bring the score even > > higher. > > > > On 04/18/2016 07:33 PM, Doug Hellmann wrote: > > > Can you truly not imagine any other useful way to package OpenStack > > > other than individual packages with shared dependencies that would > > > be acceptable? > > > > I could imagine that, but that's not at all what OpenStack is currently. > > Currently, each and every service is using Oslo, clients, and so on, > > plus the glue libs like os-brick, Castellan, you name it. > > > > Or did I misunderstood your sentence above? > > > > On 04/18/2016 07:49 PM, Sean Dague wrote: > > > A lot of distros specifically have policies against this kind of > > > bundling as well, because of security issues like this (which was so > > > very bad) - http://www.zlib.net/advisory-2002-03-11.txt > > > > Gentoo is probably a special case (because building from source and so > > on), but I'd say that *ALL* distro do have such policy in place (at > > least, Red Hat, Debian, Ubuntu, SuSE...). > > > > On 04/18/2016 08:30 PM, Doug Hellmann wrote: > > > Our upstream > > > solution can be pretty light-weight, and that leaves room for > > > downstream folks to make different choices. > > > > I'm not sure what choice we'd have, but to face loads of issues and > > finally give-up packaging OpenStack if we have python module versions > > that are conflicting. Hopefully, your proposal wont allow that? > > > > On 04/18/2016 08:40 PM, Doug Hellmann wrote: > > > Some folks seem to be conflating "upstream wants to stop worrying > > > about co-installability" with "upstream wants us to bundle > > > dependencies". That's not what I'm proposing. > > > > Well, if we need multiple versions of the same Python module, what is > > the solution in production then? > > > > > Another alternate that might work is for downstream folks to do > > > their own dependency management. > > > > I fail to understand what you think distros could do to fix the > > situation of conflicting versions, other than bundling, which isn't > > acceptable. Could you explain what you mean by "downstream folks to do > > their own dependency management"? > > > > > That's less optimal, and may not > > > change the outcome for Gentoo or Debian, where the downstream teams > > > are small (1 person each, I think?). > > > > You can also add "Ubuntu" in the list here, as absolutely all OpenStack > > dependencies are maintained mostly by me, within Debian, and then later > > "absolutely all" of OpenStack's dependencies are not maintained by you in > Debian. A significant number are maintained by the DPMT (Debian Python > Modules Team). The large majority are maintained by you, but not > "absolutely all". > > > synced to Ubuntu, with a bit of help from Ubuntu folks (but they are > And if you grep the changelogs for openstack deps in Debian you'd see a lot more than "a bit" of contributions from Ubuntu folks. But that's not the point of this thread.. > mostly busy with Canonical own products, and I'm the person doing most > > of the work, still). This is slowly shifting to a more spread workload, > > but unfortunately, we're not there yet. > > > > As for Red Hat / RDO, Haikel told me they are "2 and a half", so > > basically, the situation there (unless it changed) is approximately the > > same amount of burnout people. > > > > > I started the discussion to solicit ideas, but I would prefer to > > > avoid proposing what downstream should do because (a) I'm sure > > > folks want options and (b) I'm sure I'm not the person to identify > > > those options. > > > > Well, I'm doing the packaging, and I'm not sure if we have any option > > either, if 2 python dependencies are conflicting. > > > > On 04/18/2016 09:10 PM, Jeremy Stanley wrote: > > > On 2016-04-18 13:58:03 -0500 (-0500), Matthew Thode wrote: > > >> Ya, I'd be happy to work more with upstream. I already review the > > >> stable-reqs updates and watch them for the stable branches I package > > >> for. Not sure what else is needed. > > > > > > Reviewing the master branch openstack/requirements repository > > > changes (to make sure deps being added are going to be sane things > > > for someone in your distro to maintain packages of in the long term) > > > would also make sense. > > > > I've done some of that, and used to be reactive on it. But mostly, the > > way it's been done so far is very good (thanks to a lot of folks working > > on it), and it's taking too much of my time to review them. So I just > > pick it up before packaging a (beta-)version of OpenStack. > > > > > To be clear, "Moving the burden to packagers" is not the only option > > > available to us. I've proposed one option for eliminating the issue, > > > which has some benefits for us upstream but obviously introduces > > > some other issues we would need to resolve. Another option is for > > > more people to get involved in managing the dependency list. Some > > > (most? all?) of those new people may come from distros, and sharing > > > the effort among them would make it easier than each of them doing > > > all of the work individually. Sort of like an open source project. > > > > There's 2 options I see to solve the "people reviewing the > > global-requirements.txt changes aren't enough". > > > > 1/ talk to my boss and the folks from Canonical (I can't tell for the > > RPM world) to allocate more staff for packaging (yes, I'm serious here! > > I can get you in touch...). > > > > 2/ get a lot of dependencies out. Yes, let's really do that! We're > > constantly adding more, I've seen only very few going away. Currently > > the global-requirements.txt for stable/mitaka is 377 lines long. If you > > grep out "client", "oslo" and "os-", "openstack", comments, empty lines, > > and a few others, there's about 260 dependencies left. That's insane at > > multiple levels. When are we going to see a movement to clean this mess, > > and define real OpenStack standards? > > > > Cheers, > > > > Thomas Goirand (zigo) > > > > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > -- > Ian Cordasco > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Regards, Corey
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
