You’re aware pip has a built in command to create a local cache of wheels yea?
Not sure what your requirements are or what you’re using it for but if you detail your requirements I can tell you how well pip can handle the use case out of the box. On Jul 9, 2014, at 12:19 PM, Ben Nemec <openst...@nemebean.com> wrote: > On 07/08/2014 11:05 PM, Joe Gordon wrote: >> On Tue, Jul 8, 2014 at 8:54 PM, James Polley <j...@jamezpolley.com> wrote: >> >>> It may not have been clear from the below email, but clarkb clarifies on >>> https://bugs.launchpad.net/openstack-ci/+bug/1294381 that the infra team >>> is no longer maintaining pypi-mirror >>> >>> This has been a very useful tool for tripleo. It's much simpler for new >>> developers to set up and use than a full bandersnatch mirror (and requires >>> less disk space), and it can create a local cache of wheels which saves >>> build time. >>> >>> But it's now unsupported. >>> >>> To me it seems like we have two options: >>> >>> A) Deprecate usage of pypi-mirror; update docs to instruct new devs in >>> setting up a local bandersnatch mirror instead >>> or >>> B) Take on care-and-feeding of the tool. >>> or, I guess, >>> C) Continue to recommend people use an unsupported unmaintained >>> known-buggy tool (it works reasonably well for us today, but it's going to >>> work less and less well as time goes by) >>> >>> Are there other options I haven't thought of? >>> >> >> I don't know if this fits your requirements but I use >> http://doc.devpi.net/latest/quickstart-pypimirror.html for my development >> needs. > > Will that also cache wheels? In my experience, wheels are one of the > big time savers in tripleo so I would consider it an important feature > to maintain, however we decide to proceed. > >> >> >>> >>> Do you have thoughts on which option is preferred? >>> >>> >>> ---------- Forwarded message ---------- >>> From: Clark Boylan <clark.boy...@gmail.com> >>> Date: Tue, Jul 8, 2014 at 8:50 AM >>> Subject: Re: [openstack-dev] Policy around Requirements Adds (was: New >>> class of requirements for Stackforge projects) >>> To: "OpenStack Development Mailing List (not for usage questions)" < >>> openstack-dev@lists.openstack.org> >>> >>> >>> On Mon, Jul 7, 2014 at 3:45 PM, Joe Gordon <joe.gord...@gmail.com> wrote: >>>> >>>> On Jul 7, 2014 4:48 PM, "Sean Dague" <s...@dague.net> wrote: >>>>> >>>>> This thread was unfortunately hidden under a project specific tag (I >>>>> have thus stripped all the tags). >>>>> >>>>> The crux of the argument here is the following: >>>>> >>>>> Is a stackforge project project able to propose additions to >>>>> global-requirements.txt that aren't used by any projects in OpenStack. >>>>> >>>>> I believe the answer is firmly *no*. >>>> >>>> ++ >>>> >>>>> >>>>> global-requirements.txt provides a way for us to have a single point of >>>>> vetting for requirements for OpenStack. It lets us assess licensing, >>>>> maturity, current state of packaging, python3 support, all in one place. >>>>> And it lets us enforce that integration of OpenStack projects all run >>>>> under a well understood set of requirements. >>>>> >>>>> The requirements sync that happens after requirements land is basically >>>>> just a nicety for getting openstack projects to the tested state by >>>>> eventual consistency. >>>>> >>>>> If a stackforge project wants to be limited by global-requirements, >>>>> that's cool. We have a mechanism for that. However, they are accepting >>>>> that they will be limited by it. That means they live with how the >>>>> OpenStack project establishes that list. It specifically means they >>>>> *don't* get to propose any new requirements. >>>>> >>>>> Basically in this case Solum wants to have it's cake and eat it to. Both >>>>> be enforced on requirements, and not be enforced. Or some 3rd thing that >>>>> means the same as that. >>>>> >>>>> The near term fix is to remove solum from projects.txt. >>>> >>>> The email included below mentions an additional motivation for using >>>> global-requirements is to avoid using pypi.python.org and instead use >>>> pypi.openstack.org for speed and reliability. Perhaps there is a way we >>> can >>>> support use case for stackforge projects not in projects.txt? I thought I >>>> saw something the other day about adding a full pypi mirror to OpenStack >>>> infra. >>>> >>> This is done. All tests are now run against a bandersnatch built full >>> mirror of pypi. Enforcement of the global requirements is performed >>> via the enforcement jobs. >>>>> >>>>> On 06/26/2014 02:00 AM, Adrian Otto wrote: >>>>>> Ok, >>>>>> >>>>>> I submitted and abandoned a couple of reviews[1][2] for a solution >>> aimed >>>>>> to meet my goals without adding a new per-project requirements file. >>> The >>>>>> flaw with this approach is that pip may install other requirements >>> when >>>>>> installing the one(s) loaded from the fallback mirror, and those may >>>>>> conflict with the ones loaded from the primary mirror. >>>>>> >>>>>> After discussing this further in #openstack-infra this evening, we >>>>>> should give serious consideration to adding python-mistralclient to >>>>>> global requirements. I have posted a review[3] for that to get input >>>>>> from the requirements review team. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Adrian >>>>>> >>>>>> [1] https://review.openstack.org/102716 >>>>>> [2] https://review.openstack.org/102719 >>>>>> [3] https://review.openstack.org/102738 >>>>>> <https://review.openstack.org/1027387> >>>>>> >>>>>> On Jun 25, 2014, at 9:51 PM, Matthew Oliver <m...@oliver.net.au >>>>>> <mailto:m...@oliver.net.au>> wrote: >>>>>> >>>>>>> >>>>>>> On Jun 26, 2014 12:12 PM, "Angus Salkeld" < >>> angus.salk...@rackspace.com >>>>>>> <mailto:angus.salk...@rackspace.com>> wrote: >>>>>>>> >>>>>> On 25/06/14 15:13, Clark Boylan wrote: >>>>>>> On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto >>>>>>>> <adrian.o...@rackspace.com <mailto:adrian.o...@rackspace.com>> >>> wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> Solum has run into a constraint with the current scheme for >>>>>>>> requirements management within the OpenStack CI system. We have a >>>>>>>> proposal for dealing with this constraint that involves making a >>>>>>>> contribution to openstack-infra. This message explains the >>> constraint, >>>>>>>> and our proposal for addressing it. >>>>>>>> >>>>>>>> == Background == >>>>>>>> >>>>>>>> OpenStack uses a list of global requirements in the requirements >>>>>>>> repo[1], and each project has it’s own requirements.txt and >>>>>>>> test-requirements.txt files. The requirements are satisfied by gate >>>>>>>> jobs using pip configured to use the pypi.openstack.org >>>>>>>> <http://pypi.openstack.org/> mirror, which is periodically updated >>>>>>>> with new content from pypi.python.org <http://pypi.python.org/>. >>> One >>>>>>>> motivation for doing this is that pypi.python.org >>>>>>>> <http://pypi.python.org/> may not be as fast or as reliable as a >>> local >>>>>>>> mirror. The gate/check jobs for the projects use the OpenStack >>>>>>>> internal pypi mirror to ensure stability. >>>>>>>> >>>>>>>> The OpenStack CI system will sync up the requirements across all >>>>>>>> the official projects and will create reviews in the participating >>>>>>>> projects for any mis-matches. Solum is one of these projects, and >>>>>>>> enjoys this feature. >>>>>>>> >>>>>>>> Another motivation is so that users of OpenStack will have one >>>>>>>> single set of python package requirements/dependencies to install >>> and >>>>>>>> run the individual OpenStack components. >>>>>>>> >>>>>>>> == Problem == >>>>>>>> >>>>>>>> Stackforge projects listed in openstack/requirements/projects.txt >>>>>>>> that decide to depend on each other (for example, Solum wanting to >>>>>>>> list mistralclient as a requirement) are unable to, because they are >>>>>>>> not yet integrated, and are not listed in >>>>>>>> openstack/requirements/global-requirements.txt yet. This means that >>> in >>>>>>>> order to depend on each other, a project must withdraw from >>>>>>>> projects.txt and begin using pip with pypi.poython.org >>>>>>>> <http://pypi.poython.org/> to satisfy all of their requirements.I >>>>>>>> strongly dislike this option. >>>>>>>> >>>>>>>> Mistral is still evolving rapidly, and we don’t think it makes >>>>>>>> sense for them to pursue integration wight now. The upstream >>>>>>>> distributions who include packages to support OpenStack will also >>>>>>>> prefer not to deal with a requirement that will be cutting a new >>>>>>>> version every week or two in order to satisfy evolving needs as >>> Solum >>>>>>>> and other consumers of Mistral help refine how it works. >>>>>>>> >>>>>>>> == Proposal == >>>>>>>> >>>>>>>> We want the best of both worlds. We want the freedom to innovate >>>>>>>> and use new software for a limited selection of stackforge projects, >>>>>>>> and still use the OpenStack pypi server to satisfy my regular >>>>>>>> requirements. We want the speed and reliability of using our local >>>>>>>> mirror, and users of Solum to use a matching set of requirements for >>>>>>>> all the things that we use, and integrated projects use. We want to >>>>>>>> continue getting the reviews that bring us up to date with new >>>>>>>> requirements versions. >>>>>>>> >>>>>>>> We propose that we submit an enhancement to the gate/check job >>>>>>>> setup that will: >>>>>>>> >>>>>>>> 1) Begin (as it does today) by satisfying global-requirements.txt >>>>>>>> and my local project’s requirements.txt and test-requirements.txt >>>>>>>> using the local OpenStack pypi mirror. >>>>>>>> 2) After all requirements are satisfied, check the name of my >>>>>>>> project. If it begins with ‘stackforge/‘ then look for a >>>>>>>> stackforge-requirements.txt file. If one exists, reconfigure pip to >>>>>>>> switch to use pypi.python.org <http://pypi.python.org/>, and >>> satisfy >>>>>>>> the requirements listed in the file. We will list mistralclient >>> there, >>>>>>>> and get the latest tagged/released version of that. >>>>>>>> >>>>>>> I am reasonably sure that if you remove yourself from the >>>>>>> openstack/requirements project list this is basically how it will >>>>>>> work. Pip is configured to use the OpenStack mirror and fall back on >>>>>>> pypi.python.org <http://pypi.python.org/> for packages not >>>>>>>> available on the OpenStack mirror >>>>>>> [2]. So I don't think there is any work to do here with additional >>>>>>> requirements files. It should just work. Adding a new requirements >>>>>>> file will just make things more confusing for packagers and consumers >>>>>>> of your software. >>>>>> >>>>>> Adrian I know this is not the optimal solution, but I think this is >>>>>> the most pragmatic solution (esp. given we need to progress and not >>>>>>>> be held >>>>>> up by this), most stackforge projects are in the same boat as us. >>>>>> As far as pypi breakages (most are easily fixable by restricting the >>>>>> package versions if we get an issue with a new release >>>>>> of *random-api-breaking-package*). >>>>>> >>>>>>>> >>>>>>>> I've looked through the infra choose mirror code, and Clark is >>>>>>>> correct. If the project isn't in the projects.txt file they will >>> only >>>>>>>> access to pypi.openstack.org <http://pypi.openstack.org/> however >>> if >>>>>>>> removed then it will first check pypi.openstack.org >>>>>>>> <http://pypi.openstack.org/> and then fall back to to >>> pypi.python.org >>>>>>>> <http://pypi.python.org/>. I think the only real solution is what >>>>>>>> Angus mentioned, remove yourself from projects.txt at least until >>> all >>>>>>>> your dependencies can be provided by pypi.openstack.org >>>>>>>> <http://pypi.openstack.org/> or another solution is put into >>> place. In >>>>>>>> the mean time you can at least progress and continue development. >>>>>>>> >>>>>>>> If you code requires a direct dependency (rather then an optional >>>>>>>> dependency) of some non integrated project, then your stuck until >>> they >>>>>>>> are. >>>>>>>> >>>>>>>> >>>>>> >>>>>>>> >>>>>>>> == Call To Action == >>>>>>>> >>>>>>>> What do you think of this approach to satisfy a balance of >>>>>>>> interests? Everything remains the same for OpenStack projects, and >>>>>>>> Stackforge projects get a new feature that allows them to require >>>>>>>> software that has not yet been integrated. Are there even better >>>>>>>> options that we should consider? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Adrian Otto >>>>>>>> >>>>>>>> >>>>>>>> References: >>>>>>>> [1] https://review.openstack.org/openstack/requirements >>>>>> >>>>>>> For what it is worth the Infra team has also been looking at >>>>>>> potentially using something like bandersnatch to mirror all of pypi >>>>>>> which is now a possibility because OpenStack doesn't depend on >>>>>>> packages that are hosted external to pypi. We would then do >>>>>>> requirements enforcement via checks rather than explicit use of a >>>>>>> restricted mirror. There are some things to sort out like platform >>>>>>> dependent wheels (I am not sure that any OpenStack project directly >>>>>>> consumes these but I have found them to be quite handy) and the >>>>>>> potential need for more enforcement to keep this working, but I think >>>>>>> this is a possibility. >>>>>> >>>>>> This would be neat. >>>>>> >>>>>> -Angus >>>>>> >>>>>> >>>>>>> Clark >>>>>> >>>>>>> [2] >>>>>>>> >>>>>>>> >>> https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/slave_scripts/select-mirror.sh#n54 >>>>>> >>>>>>> _______________________________________________ >>>>>>> OpenStack-dev mailing list >>>>>>> OpenStack-dev@lists.openstack.org >>>>>>>> <mailto:OpenStack-dev@lists.openstack.org> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> OpenStack-dev mailing list >>>>>>>> OpenStack-dev@lists.openstack.org >>>>>>> <mailto:OpenStack-dev@lists.openstack.org> >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/ >>>>>>> >>>>>>> _______________________________________________ >>>>>>> OpenStack-dev mailing list >>>>>>> OpenStack-dev@lists.openstack.org >>>>>>> <mailto:OpenStack-dev@lists.openstack.org> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> OpenStack-dev mailing list >>>>>> OpenStack-dev@lists.openstack.org >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>>> >>>>> >>>>> -- >>>>> Sean Dague >>>>> http://dague.net >>>>> >>>>> >>>>> _______________________________________________ >>>>> OpenStack-dev mailing list >>>>> OpenStack-dev@lists.openstack.org >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>>> >>>> >>>> >>>> _______________________________________________ >>>> OpenStack-dev mailing list >>>> OpenStack-dev@lists.openstack.org >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>>> >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
