On 07/08/2014 11:05 PM, Joe Gordon wrote: > On Tue, Jul 8, 2014 at 8:54 PM, James Polley <j...@jamezpolley.com> wrote: > >> It may not have been clear from the below email, but clarkb clarifies on >> https://bugs.launchpad.net/openstack-ci/+bug/1294381 that the infra team >> is no longer maintaining pypi-mirror >> >> This has been a very useful tool for tripleo. It's much simpler for new >> developers to set up and use than a full bandersnatch mirror (and requires >> less disk space), and it can create a local cache of wheels which saves >> build time. >> >> But it's now unsupported. >> >> To me it seems like we have two options: >> >> A) Deprecate usage of pypi-mirror; update docs to instruct new devs in >> setting up a local bandersnatch mirror instead >> or >> B) Take on care-and-feeding of the tool. >> or, I guess, >> C) Continue to recommend people use an unsupported unmaintained >> known-buggy tool (it works reasonably well for us today, but it's going to >> work less and less well as time goes by) >> >> Are there other options I haven't thought of? >> > > I don't know if this fits your requirements but I use > http://doc.devpi.net/latest/quickstart-pypimirror.html for my development > needs.
Will that also cache wheels? In my experience, wheels are one of the big time savers in tripleo so I would consider it an important feature to maintain, however we decide to proceed. > > >> >> Do you have thoughts on which option is preferred? >> >> >> ---------- Forwarded message ---------- >> From: Clark Boylan <clark.boy...@gmail.com> >> Date: Tue, Jul 8, 2014 at 8:50 AM >> Subject: Re: [openstack-dev] Policy around Requirements Adds (was: New >> class of requirements for Stackforge projects) >> To: "OpenStack Development Mailing List (not for usage questions)" < >> openstack-dev@lists.openstack.org> >> >> >> On Mon, Jul 7, 2014 at 3:45 PM, Joe Gordon <joe.gord...@gmail.com> wrote: >>> >>> On Jul 7, 2014 4:48 PM, "Sean Dague" <s...@dague.net> wrote: >>>> >>>> This thread was unfortunately hidden under a project specific tag (I >>>> have thus stripped all the tags). >>>> >>>> The crux of the argument here is the following: >>>> >>>> Is a stackforge project project able to propose additions to >>>> global-requirements.txt that aren't used by any projects in OpenStack. >>>> >>>> I believe the answer is firmly *no*. >>> >>> ++ >>> >>>> >>>> global-requirements.txt provides a way for us to have a single point of >>>> vetting for requirements for OpenStack. It lets us assess licensing, >>>> maturity, current state of packaging, python3 support, all in one place. >>>> And it lets us enforce that integration of OpenStack projects all run >>>> under a well understood set of requirements. >>>> >>>> The requirements sync that happens after requirements land is basically >>>> just a nicety for getting openstack projects to the tested state by >>>> eventual consistency. >>>> >>>> If a stackforge project wants to be limited by global-requirements, >>>> that's cool. We have a mechanism for that. However, they are accepting >>>> that they will be limited by it. That means they live with how the >>>> OpenStack project establishes that list. It specifically means they >>>> *don't* get to propose any new requirements. >>>> >>>> Basically in this case Solum wants to have it's cake and eat it to. Both >>>> be enforced on requirements, and not be enforced. Or some 3rd thing that >>>> means the same as that. >>>> >>>> The near term fix is to remove solum from projects.txt. >>> >>> The email included below mentions an additional motivation for using >>> global-requirements is to avoid using pypi.python.org and instead use >>> pypi.openstack.org for speed and reliability. Perhaps there is a way we >> can >>> support use case for stackforge projects not in projects.txt? I thought I >>> saw something the other day about adding a full pypi mirror to OpenStack >>> infra. >>> >> This is done. All tests are now run against a bandersnatch built full >> mirror of pypi. Enforcement of the global requirements is performed >> via the enforcement jobs. >>>> >>>> On 06/26/2014 02:00 AM, Adrian Otto wrote: >>>>> Ok, >>>>> >>>>> I submitted and abandoned a couple of reviews[1][2] for a solution >> aimed >>>>> to meet my goals without adding a new per-project requirements file. >> The >>>>> flaw with this approach is that pip may install other requirements >> when >>>>> installing the one(s) loaded from the fallback mirror, and those may >>>>> conflict with the ones loaded from the primary mirror. >>>>> >>>>> After discussing this further in #openstack-infra this evening, we >>>>> should give serious consideration to adding python-mistralclient to >>>>> global requirements. I have posted a review[3] for that to get input >>>>> from the requirements review team. >>>>> >>>>> Thanks, >>>>> >>>>> Adrian >>>>> >>>>> [1] https://review.openstack.org/102716 >>>>> [2] https://review.openstack.org/102719 >>>>> [3] https://review.openstack.org/102738 >>>>> <https://review.openstack.org/1027387> >>>>> >>>>> On Jun 25, 2014, at 9:51 PM, Matthew Oliver <m...@oliver.net.au >>>>> <mailto:m...@oliver.net.au>> wrote: >>>>> >>>>>> >>>>>> On Jun 26, 2014 12:12 PM, "Angus Salkeld" < >> angus.salk...@rackspace.com >>>>>> <mailto:angus.salk...@rackspace.com>> wrote: >>>>>>> >>>>> On 25/06/14 15:13, Clark Boylan wrote: >>>>>> On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto >>>>>>> <adrian.o...@rackspace.com <mailto:adrian.o...@rackspace.com>> >> wrote: >>>>>>> Hello, >>>>>>> >>>>>>> Solum has run into a constraint with the current scheme for >>>>>>> requirements management within the OpenStack CI system. We have a >>>>>>> proposal for dealing with this constraint that involves making a >>>>>>> contribution to openstack-infra. This message explains the >> constraint, >>>>>>> and our proposal for addressing it. >>>>>>> >>>>>>> == Background == >>>>>>> >>>>>>> OpenStack uses a list of global requirements in the requirements >>>>>>> repo[1], and each project has it’s own requirements.txt and >>>>>>> test-requirements.txt files. The requirements are satisfied by gate >>>>>>> jobs using pip configured to use the pypi.openstack.org >>>>>>> <http://pypi.openstack.org/> mirror, which is periodically updated >>>>>>> with new content from pypi.python.org <http://pypi.python.org/>. >> One >>>>>>> motivation for doing this is that pypi.python.org >>>>>>> <http://pypi.python.org/> may not be as fast or as reliable as a >> local >>>>>>> mirror. The gate/check jobs for the projects use the OpenStack >>>>>>> internal pypi mirror to ensure stability. >>>>>>> >>>>>>> The OpenStack CI system will sync up the requirements across all >>>>>>> the official projects and will create reviews in the participating >>>>>>> projects for any mis-matches. Solum is one of these projects, and >>>>>>> enjoys this feature. >>>>>>> >>>>>>> Another motivation is so that users of OpenStack will have one >>>>>>> single set of python package requirements/dependencies to install >> and >>>>>>> run the individual OpenStack components. >>>>>>> >>>>>>> == Problem == >>>>>>> >>>>>>> Stackforge projects listed in openstack/requirements/projects.txt >>>>>>> that decide to depend on each other (for example, Solum wanting to >>>>>>> list mistralclient as a requirement) are unable to, because they are >>>>>>> not yet integrated, and are not listed in >>>>>>> openstack/requirements/global-requirements.txt yet. This means that >> in >>>>>>> order to depend on each other, a project must withdraw from >>>>>>> projects.txt and begin using pip with pypi.poython.org >>>>>>> <http://pypi.poython.org/> to satisfy all of their requirements.I >>>>>>> strongly dislike this option. >>>>>>> >>>>>>> Mistral is still evolving rapidly, and we don’t think it makes >>>>>>> sense for them to pursue integration wight now. The upstream >>>>>>> distributions who include packages to support OpenStack will also >>>>>>> prefer not to deal with a requirement that will be cutting a new >>>>>>> version every week or two in order to satisfy evolving needs as >> Solum >>>>>>> and other consumers of Mistral help refine how it works. >>>>>>> >>>>>>> == Proposal == >>>>>>> >>>>>>> We want the best of both worlds. We want the freedom to innovate >>>>>>> and use new software for a limited selection of stackforge projects, >>>>>>> and still use the OpenStack pypi server to satisfy my regular >>>>>>> requirements. We want the speed and reliability of using our local >>>>>>> mirror, and users of Solum to use a matching set of requirements for >>>>>>> all the things that we use, and integrated projects use. We want to >>>>>>> continue getting the reviews that bring us up to date with new >>>>>>> requirements versions. >>>>>>> >>>>>>> We propose that we submit an enhancement to the gate/check job >>>>>>> setup that will: >>>>>>> >>>>>>> 1) Begin (as it does today) by satisfying global-requirements.txt >>>>>>> and my local project’s requirements.txt and test-requirements.txt >>>>>>> using the local OpenStack pypi mirror. >>>>>>> 2) After all requirements are satisfied, check the name of my >>>>>>> project. If it begins with ‘stackforge/‘ then look for a >>>>>>> stackforge-requirements.txt file. If one exists, reconfigure pip to >>>>>>> switch to use pypi.python.org <http://pypi.python.org/>, and >> satisfy >>>>>>> the requirements listed in the file. We will list mistralclient >> there, >>>>>>> and get the latest tagged/released version of that. >>>>>>> >>>>>> I am reasonably sure that if you remove yourself from the >>>>>> openstack/requirements project list this is basically how it will >>>>>> work. Pip is configured to use the OpenStack mirror and fall back on >>>>>> pypi.python.org <http://pypi.python.org/> for packages not >>>>>>> available on the OpenStack mirror >>>>>> [2]. So I don't think there is any work to do here with additional >>>>>> requirements files. It should just work. Adding a new requirements >>>>>> file will just make things more confusing for packagers and consumers >>>>>> of your software. >>>>> >>>>> Adrian I know this is not the optimal solution, but I think this is >>>>> the most pragmatic solution (esp. given we need to progress and not >>>>>>> be held >>>>> up by this), most stackforge projects are in the same boat as us. >>>>> As far as pypi breakages (most are easily fixable by restricting the >>>>> package versions if we get an issue with a new release >>>>> of *random-api-breaking-package*). >>>>> >>>>>>> >>>>>>> I've looked through the infra choose mirror code, and Clark is >>>>>>> correct. If the project isn't in the projects.txt file they will >> only >>>>>>> access to pypi.openstack.org <http://pypi.openstack.org/> however >> if >>>>>>> removed then it will first check pypi.openstack.org >>>>>>> <http://pypi.openstack.org/> and then fall back to to >> pypi.python.org >>>>>>> <http://pypi.python.org/>. I think the only real solution is what >>>>>>> Angus mentioned, remove yourself from projects.txt at least until >> all >>>>>>> your dependencies can be provided by pypi.openstack.org >>>>>>> <http://pypi.openstack.org/> or another solution is put into >> place. In >>>>>>> the mean time you can at least progress and continue development. >>>>>>> >>>>>>> If you code requires a direct dependency (rather then an optional >>>>>>> dependency) of some non integrated project, then your stuck until >> they >>>>>>> are. >>>>>>> >>>>>>> >>>>> >>>>>>> >>>>>>> == Call To Action == >>>>>>> >>>>>>> What do you think of this approach to satisfy a balance of >>>>>>> interests? Everything remains the same for OpenStack projects, and >>>>>>> Stackforge projects get a new feature that allows them to require >>>>>>> software that has not yet been integrated. Are there even better >>>>>>> options that we should consider? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Adrian Otto >>>>>>> >>>>>>> >>>>>>> References: >>>>>>> [1] https://review.openstack.org/openstack/requirements >>>>> >>>>>> For what it is worth the Infra team has also been looking at >>>>>> potentially using something like bandersnatch to mirror all of pypi >>>>>> which is now a possibility because OpenStack doesn't depend on >>>>>> packages that are hosted external to pypi. We would then do >>>>>> requirements enforcement via checks rather than explicit use of a >>>>>> restricted mirror. There are some things to sort out like platform >>>>>> dependent wheels (I am not sure that any OpenStack project directly >>>>>> consumes these but I have found them to be quite handy) and the >>>>>> potential need for more enforcement to keep this working, but I think >>>>>> this is a possibility. >>>>> >>>>> This would be neat. >>>>> >>>>> -Angus >>>>> >>>>> >>>>>> Clark >>>>> >>>>>> [2] >>>>>>> >>>>>>> >> https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/slave_scripts/select-mirror.sh#n54 >>>>> >>>>>> _______________________________________________ >>>>>> OpenStack-dev mailing list >>>>>> OpenStack-dev@lists.openstack.org >>>>>>> <mailto:OpenStack-dev@lists.openstack.org> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> OpenStack-dev mailing list >>>>>>> OpenStack-dev@lists.openstack.org >>>>>> <mailto:OpenStack-dev@lists.openstack.org> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/ >>>>>> >>>>>> _______________________________________________ >>>>>> OpenStack-dev mailing list >>>>>> OpenStack-dev@lists.openstack.org >>>>>> <mailto:OpenStack-dev@lists.openstack.org> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> OpenStack-dev mailing list >>>>> OpenStack-dev@lists.openstack.org >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>> >>>> -- >>>> Sean Dague >>>> http://dague.net >>>> >>>> >>>> _______________________________________________ >>>> OpenStack-dev mailing list >>>> OpenStack-dev@lists.openstack.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>> >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
