On Thu, Feb 19, 2015 at 1:48 PM, Adam Gandelman <ad...@ubuntu.com> wrote:
> This creates a bit of a problem for downstream (packagers and probably > others) Shipping a requirements.txt with explicit pins will end up > producing an egg with a requires.txt that reflects those pins, unless there > is some other magic planned that I'm not aware of. I can't speak for all > packaging flavors, but I know debian packaging interacts quite closely with > things like requirements.txt and resulting egg's requires.txt to determine > appropriate system-level package dependencies. This would require a lot of > tedious work on packagers part to get something functional. > > What if its flipped? How about keeping requirements.txt with the caps, and > using that as input to produce something like requirements.gate that passed > to 'pip install --no-deps' on our slaves? We'd end up installing and > using the explicit pinned requirements while the services/libraries > themselves remain flexible. This might the issue Doug pointed out, where > requirements updates across projects are not synchronized. > > Switching them to requirements.txt and requirements.gate works for me. If a simple renaming makes things better, then great! As for Doug's comment, yes we need to work something out to overwrite requirements.gate, under your proposed naming, with global requirments > Adam > > > > On Thu, Feb 19, 2015 at 12:59 PM, Joe Gordon <joe.gord...@gmail.com> > wrote: > >> >> >> On Wed, Feb 18, 2015 at 7:14 AM, Doug Hellmann <d...@doughellmann.com> >> wrote: >> >>> >>> >>> On Wed, Feb 18, 2015, at 10:07 AM, Donald Stufft wrote: >>> > >>> > > On Feb 18, 2015, at 10:00 AM, Doug Hellmann <d...@doughellmann.com> >>> wrote: >>> > > >>> > > >>> > > >>> > > On Tue, Feb 17, 2015, at 03:17 PM, Joe Gordon wrote: >>> > >> On Tue, Feb 17, 2015 at 4:19 AM, Sean Dague <s...@dague.net> wrote: >>> > >> >>> > >>> On 02/16/2015 08:50 PM, Ian Cordasco wrote: >>> > >>>> On 2/16/15, 16:08, "Sean Dague" <s...@dague.net> wrote: >>> > >>>> >>> > >>>>> On 02/16/2015 02:08 PM, Doug Hellmann wrote: >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> On Mon, Feb 16, 2015, at 01:01 PM, Ian Cordasco wrote: >>> > >>>>>>> Hey everyone, >>> > >>>>>>> >>> > >>>>>>> The os-ansible-deployment team was working on updates to add >>> support >>> > >>>>>>> for >>> > >>>>>>> the latest version of juno and noticed some interesting version >>> > >>>>>>> specifiers >>> > >>>>>>> introduced into global-requirements.txt in January. It >>> introduced some >>> > >>>>>>> version specifiers that seem a bit impossible like the one for >>> > >>> requests >>> > >>>>>>> [1]. There are others that equate presently to pinning the >>> versions of >>> > >>>>>>> the >>> > >>>>>>> packages [2, 3, 4]. >>> > >>>>>>> >>> > >>>>>>> I understand fully and support the commit because of how it >>> improves >>> > >>>>>>> pretty much everyone’s quality of life (no fires to put out in >>> the >>> > >>>>>>> middle >>> > >>>>>>> of the night on the weekend). I’m also aware that a lot of the >>> > >>>>>>> downstream >>> > >>>>>>> redistributors tend to work from global-requirements.txt when >>> > >>>>>>> determining >>> > >>>>>>> what to package/support. >>> > >>>>>>> >>> > >>>>>>> It seems to me like there’s room to clean up some of these >>> > >>> requirements >>> > >>>>>>> to >>> > >>>>>>> make them far more explicit and less misleading to the human >>> eye (even >>> > >>>>>>> though tooling like pip can easily parse/understand these). >>> > >>>>>> >>> > >>>>>> I think that's the idea. These requirements were generated >>> > >>>>>> automatically, and fixed issues that were holding back several >>> > >>> projects. >>> > >>>>>> Now we can apply updates to them by hand, to either move the >>> lower >>> > >>>>>> bounds down (as in the case Ihar pointed out with stevedore) or >>> clean >>> > >>> up >>> > >>>>>> the range definitions. We should not raise the limits of any >>> Oslo >>> > >>>>>> libraries, and we should consider raising the limits of >>> third-party >>> > >>>>>> libraries very carefully. >>> > >>>>>> >>> > >>>>>> We should make those changes on one library at a time, so we >>> can see >>> > >>>>>> what effect each change has on the other requirements. >>> > >>>>>> >>> > >>>>>>> >>> > >>>>>>> I also understand that stable-maint may want to occasionally >>> bump the >>> > >>>>>>> caps >>> > >>>>>>> to see if newer versions will not break everything, so what is >>> the >>> > >>>>>>> right >>> > >>>>>>> way forward? What is the best way to both maintain a stable >>> branch >>> > >>> with >>> > >>>>>>> known working dependencies while helping out those who do so >>> much work >>> > >>>>>>> for >>> > >>>>>>> us (downstream and stable-maint) and not permanently pinning to >>> > >>> certain >>> > >>>>>>> working versions? >>> > >>>>>> >>> > >>>>>> Managing the upper bounds is still under discussion. Sean >>> pointed out >>> > >>>>>> that we might want hard caps so that updates to stable branch >>> were >>> > >>>>>> explicit. I can see either side of that argument and am still >>> on the >>> > >>>>>> fence about the best approach. >>> > >>>>> >>> > >>>>> History has shown that it's too much work keeping testing >>> functioning >>> > >>>>> for stable branches if we leave dependencies uncapped. If >>> particular >>> > >>>>> people are interested in bumping versions when releases happen, >>> it's >>> > >>>>> easy enough to do with a requirements proposed update. It will >>> even run >>> > >>>>> tests that in most cases will prove that it works. >>> > >>>>> >>> > >>>>> It might even be possible for someone to build some automation >>> that did >>> > >>>>> that as stuff from pypi released so we could have the best of >>> both >>> > >>>>> worlds. But I think capping is definitely something we want as a >>> > >>>>> project, and it reflects the way that most deployments will >>> consume this >>> > >>>>> code. >>> > >>>>> >>> > >>>>> -Sean >>> > >>>>> >>> > >>>>> -- >>> > >>>>> Sean Dague >>> > >>>>> http://dague.net >>> > >>>> >>> > >>>> Right. No one is arguing the very clear benefits of all of this. >>> > >>>> >>> > >>>> I’m just wondering if for the example version identifiers that I >>> gave in >>> > >>>> my original message (and others that are very similar) if we want >>> to make >>> > >>>> the strings much simpler for people who tend to work from them >>> (i.e., >>> > >>>> downstream re-distributors whose jobs are already difficult >>> enough). I’ve >>> > >>>> offered to help at least one of them in the past who maintains >>> all of >>> > >>>> their distro’s packages themselves, but they refused so I’d like >>> to help >>> > >>>> them anyway possible. Especially if any of them chime in as this >>> being >>> > >>>> something that would be helpful. >>> > >>> >>> > >>> Ok, your links got kind of scrambled. Can you next time please >>> inline >>> > >>> the key relevant content in the email, because I think we all >>> missed the >>> > >>> original message intent as the key content was only in footnotes. >>> > >>> >>> > >>> From my point of view, normalization patches would be fine. >>> > >>> >>> > >>> requests>=1.2.1,!=2.4.0,<=2.2.1 >>> > >>> >>> > >>> Is actually an odd one, because that's still there because we're >>> using >>> > >>> Trusty level requests in the tests, and my ability to have >>> devstack not >>> > >>> install that has thus far failed. >>> > >>> >>> > >>> Things like: >>> > >>> >>> > >>> osprofiler>=0.3.0,<=0.3.0 # Apache-2.0 >>> > >>> >>> > >>> Can clearly be normalized to osprofiler==0.3.0 if you want to >>> propose >>> > >>> the patch manually. >>> > >>> >>> > >> >>> > >> global-requirements for stable branches serves two uses: >>> > >> >>> > >> 1. Specify the set of dependencies that we would like to test >>> against >>> > >> 2. A tool for downstream packagers to use when determining what to >>> > >> package/support. >>> > >> >>> > >> For #1, Ideally we would like a set of all dependencies, including >>> > >> transitive, with explicit versions (very similar to the output of >>> > >> pip-freeze). But for #2 the standard requirement file with a range >>> is >>> > >> preferred. Putting an upper bound on each dependency, instead of >>> using a >>> > >> '==' was a compromise between the two use cases. >>> > >> >>> > >> Going forward I propose we have a requirements.in and a >>> requirements.txt >>> > >> file. The requirements.in file would contain the range of >>> dependencies, >>> > >> and >>> > >> requirements.txt would contain the pinned set, and eventually the >>> pinned >>> > >> set including transitive dependencies. >>> > >> >>> > >> Thoughts? >>> > > >>> > > I'm interested in seeing what that list looks like. I suspect we have >>> > > some libraries listed in the global requirements now that aren't >>> > > actually used, and I'm sure there is a long list of transitive >>> > > dependencies to add to it. >>> > > >>> > > I'm not entirely comfortable with the idea of pinning completely, >>> but I >>> > > guess it's the best of two bad options. It solves the "we don't have >>> > > enough people around to manage stable branches" problem in one way >>> (by >>> > > not letting releases outside our control break our test jobs), but >>> if we >>> > > don't have people around now to fix things who is going to keep up >>> with >>> > > updating that requirements list as new versions of projects come >>> out? We >>> > > can write a job to automatically detect new packages and test them, >>> but >>> > > who is going to review patches submitted by that bot? Maybe that's a >>> > > small enough amount of work that it will be easier to find help. >>> > > >>> > > We've been playing whack-a-mole with issues because we made changes >>> to >>> > > the way we deal with versions and dependencies without fully >>> > > understanding the consequences of some of the changes. They looked >>> > > innocent at first, but because of assumptions in other jobs or other >>> > > parts of the system they caused problems. So I think we should be >>> > > careful about making this decision and think about some of the other >>> > > things that might fall out before pushing more changes up. >>> > > >>> > > For example, if we're syncing requirements into stable branches of >>> > > projects based on requirements.txt, and that becomes a set of pins >>> > > instead of a set of ranges with caps, how do we update projects? >>> Should >>> > > we sync from requirements.in instead of requirements.txt, to allow >>> > > projects to maintain the ranges in their own requirements files? Or >>> do >>> > > we want those requirements files to reflect the pins from the global >>> > > list? >>> > >>> > I'm not sure I fully understand what folks are proposing here with two >>> > different files, but if you’re putting ``==`` specifiers into the >>> > install_requires of various projects, then I believe that is going to >>> > cause a >>> > fairly large amount of pain. >>> >>> As I understand it, Joe's idea is to have an input file >>> ("requirements.in") that uses >=, <=, and != to specify a range of valid >>> versions. Those ranges would guide packagers about what requirements we >>> think work. The list would also be "compiled" into a list of >>> requirements using only == to create a requirements.txt file thatwould >>> be used for the tests on our CI systems. >>> >> >> That is correct. The workflow would be: >> >> When preparing a stable branch: >> >> In global requirements >> >> * rename requirements.tx to requirements.in >> * create a new requirements.txt file that consists of a full set of >> pinned, that is using '==', dependencies (transitive and all). >> >> For each project consuming global requirements >> >> * rename requirements.txt to requirements.in >> * create a new requirements.txt file that consists of a full set of >> pinned, that is using '==', dependencies (transitive and all). While making >> sure the versions allign with global requirements >> * Change how we install dependencies from 'pip install -r requirements' >> to 'pip install --no-deps -r requirements.txt >> >> >> So now we have two files for requirements. requirements.txt to specify >> the exact versions we use for testing, and requirements.in to specify >> the range we think *should* work. >> >> >>> > >>> > --- >>> > Donald Stufft >>> > PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >>> > >>> > >>> __________________________________________________________________________ >>> > OpenStack Development Mailing List (not for usage questions) >>> > Unsubscribe: >>> > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> > Email had 1 attachment: >>> > + signature.asc >>> > 1k (application/pgp-signature) >>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
