On Mon, Jul 7, 2014 at 3:45 PM, Joe Gordon <joe.gord...@gmail.com> wrote: > > On Jul 7, 2014 4:48 PM, "Sean Dague" <s...@dague.net> wrote: >> >> This thread was unfortunately hidden under a project specific tag (I >> have thus stripped all the tags). >> >> The crux of the argument here is the following: >> >> Is a stackforge project project able to propose additions to >> global-requirements.txt that aren't used by any projects in OpenStack. >> >> I believe the answer is firmly *no*. > > ++ > >> >> global-requirements.txt provides a way for us to have a single point of >> vetting for requirements for OpenStack. It lets us assess licensing, >> maturity, current state of packaging, python3 support, all in one place. >> And it lets us enforce that integration of OpenStack projects all run >> under a well understood set of requirements. >> >> The requirements sync that happens after requirements land is basically >> just a nicety for getting openstack projects to the tested state by >> eventual consistency. >> >> If a stackforge project wants to be limited by global-requirements, >> that's cool. We have a mechanism for that. However, they are accepting >> that they will be limited by it. That means they live with how the >> OpenStack project establishes that list. It specifically means they >> *don't* get to propose any new requirements. >> >> Basically in this case Solum wants to have it's cake and eat it to. Both >> be enforced on requirements, and not be enforced. Or some 3rd thing that >> means the same as that. >> >> The near term fix is to remove solum from projects.txt. > > The email included below mentions an additional motivation for using > global-requirements is to avoid using pypi.python.org and instead use > pypi.openstack.org for speed and reliability. Perhaps there is a way we can > support use case for stackforge projects not in projects.txt? I thought I > saw something the other day about adding a full pypi mirror to OpenStack > infra. > This is done. All tests are now run against a bandersnatch built full mirror of pypi. Enforcement of the global requirements is performed via the enforcement jobs. >> >> On 06/26/2014 02:00 AM, Adrian Otto wrote: >> > Ok, >> > >> > I submitted and abandoned a couple of reviews[1][2] for a solution aimed >> > to meet my goals without adding a new per-project requirements file. The >> > flaw with this approach is that pip may install other requirements when >> > installing the one(s) loaded from the fallback mirror, and those may >> > conflict with the ones loaded from the primary mirror. >> > >> > After discussing this further in #openstack-infra this evening, we >> > should give serious consideration to adding python-mistralclient to >> > global requirements. I have posted a review[3] for that to get input >> > from the requirements review team. >> > >> > Thanks, >> > >> > Adrian >> > >> > [1] https://review.openstack.org/102716 >> > [2] https://review.openstack.org/102719 >> > [3] https://review.openstack.org/102738 >> > <https://review.openstack.org/1027387> >> > >> > On Jun 25, 2014, at 9:51 PM, Matthew Oliver <m...@oliver.net.au >> > <mailto:m...@oliver.net.au>> wrote: >> > >> >> >> >> On Jun 26, 2014 12:12 PM, "Angus Salkeld" <angus.salk...@rackspace.com >> >> <mailto:angus.salk...@rackspace.com>> wrote: >> >> > >> > On 25/06/14 15:13, Clark Boylan wrote: >> >> On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto >> >>> <adrian.o...@rackspace.com <mailto:adrian.o...@rackspace.com>> wrote: >> >>> Hello, >> >>> >> >>> Solum has run into a constraint with the current scheme for >> >>> requirements management within the OpenStack CI system. We have a >> >>> proposal for dealing with this constraint that involves making a >> >>> contribution to openstack-infra. This message explains the constraint, >> >>> and our proposal for addressing it. >> >>> >> >>> == Background == >> >>> >> >>> OpenStack uses a list of global requirements in the requirements >> >>> repo[1], and each project has it’s own requirements.txt and >> >>> test-requirements.txt files. The requirements are satisfied by gate >> >>> jobs using pip configured to use the pypi.openstack.org >> >>> <http://pypi.openstack.org/> mirror, which is periodically updated >> >>> with new content from pypi.python.org <http://pypi.python.org/>. One >> >>> motivation for doing this is that pypi.python.org >> >>> <http://pypi.python.org/> may not be as fast or as reliable as a local >> >>> mirror. The gate/check jobs for the projects use the OpenStack >> >>> internal pypi mirror to ensure stability. >> >>> >> >>> The OpenStack CI system will sync up the requirements across all >> >>> the official projects and will create reviews in the participating >> >>> projects for any mis-matches. Solum is one of these projects, and >> >>> enjoys this feature. >> >>> >> >>> Another motivation is so that users of OpenStack will have one >> >>> single set of python package requirements/dependencies to install and >> >>> run the individual OpenStack components. >> >>> >> >>> == Problem == >> >>> >> >>> Stackforge projects listed in openstack/requirements/projects.txt >> >>> that decide to depend on each other (for example, Solum wanting to >> >>> list mistralclient as a requirement) are unable to, because they are >> >>> not yet integrated, and are not listed in >> >>> openstack/requirements/global-requirements.txt yet. This means that in >> >>> order to depend on each other, a project must withdraw from >> >>> projects.txt and begin using pip with pypi.poython.org >> >>> <http://pypi.poython.org/> to satisfy all of their requirements.I >> >>> strongly dislike this option. >> >>> >> >>> Mistral is still evolving rapidly, and we don’t think it makes >> >>> sense for them to pursue integration wight now. The upstream >> >>> distributions who include packages to support OpenStack will also >> >>> prefer not to deal with a requirement that will be cutting a new >> >>> version every week or two in order to satisfy evolving needs as Solum >> >>> and other consumers of Mistral help refine how it works. >> >>> >> >>> == Proposal == >> >>> >> >>> We want the best of both worlds. We want the freedom to innovate >> >>> and use new software for a limited selection of stackforge projects, >> >>> and still use the OpenStack pypi server to satisfy my regular >> >>> requirements. We want the speed and reliability of using our local >> >>> mirror, and users of Solum to use a matching set of requirements for >> >>> all the things that we use, and integrated projects use. We want to >> >>> continue getting the reviews that bring us up to date with new >> >>> requirements versions. >> >>> >> >>> We propose that we submit an enhancement to the gate/check job >> >>> setup that will: >> >>> >> >>> 1) Begin (as it does today) by satisfying global-requirements.txt >> >>> and my local project’s requirements.txt and test-requirements.txt >> >>> using the local OpenStack pypi mirror. >> >>> 2) After all requirements are satisfied, check the name of my >> >>> project. If it begins with ‘stackforge/‘ then look for a >> >>> stackforge-requirements.txt file. If one exists, reconfigure pip to >> >>> switch to use pypi.python.org <http://pypi.python.org/>, and satisfy >> >>> the requirements listed in the file. We will list mistralclient there, >> >>> and get the latest tagged/released version of that. >> >>> >> >> I am reasonably sure that if you remove yourself from the >> >> openstack/requirements project list this is basically how it will >> >> work. Pip is configured to use the OpenStack mirror and fall back on >> >> pypi.python.org <http://pypi.python.org/> for packages not >> >>> available on the OpenStack mirror >> >> [2]. So I don't think there is any work to do here with additional >> >> requirements files. It should just work. Adding a new requirements >> >> file will just make things more confusing for packagers and consumers >> >> of your software. >> > >> > Adrian I know this is not the optimal solution, but I think this is >> > the most pragmatic solution (esp. given we need to progress and not >> >>> be held >> > up by this), most stackforge projects are in the same boat as us. >> > As far as pypi breakages (most are easily fixable by restricting the >> > package versions if we get an issue with a new release >> > of *random-api-breaking-package*). >> > >> >>> >> >>> I've looked through the infra choose mirror code, and Clark is >> >>> correct. If the project isn't in the projects.txt file they will only >> >>> access to pypi.openstack.org <http://pypi.openstack.org/> however if >> >>> removed then it will first check pypi.openstack.org >> >>> <http://pypi.openstack.org/> and then fall back to to pypi.python.org >> >>> <http://pypi.python.org/>. I think the only real solution is what >> >>> Angus mentioned, remove yourself from projects.txt at least until all >> >>> your dependencies can be provided by pypi.openstack.org >> >>> <http://pypi.openstack.org/> or another solution is put into place. In >> >>> the mean time you can at least progress and continue development. >> >>> >> >>> If you code requires a direct dependency (rather then an optional >> >>> dependency) of some non integrated project, then your stuck until they >> >>> are. >> >>> >> >>> >> > >> >>> >> >>> == Call To Action == >> >>> >> >>> What do you think of this approach to satisfy a balance of >> >>> interests? Everything remains the same for OpenStack projects, and >> >>> Stackforge projects get a new feature that allows them to require >> >>> software that has not yet been integrated. Are there even better >> >>> options that we should consider? >> >>> >> >>> Thanks, >> >>> >> >>> Adrian Otto >> >>> >> >>> >> >>> References: >> >>> [1] https://review.openstack.org/openstack/requirements >> > >> >> For what it is worth the Infra team has also been looking at >> >> potentially using something like bandersnatch to mirror all of pypi >> >> which is now a possibility because OpenStack doesn't depend on >> >> packages that are hosted external to pypi. We would then do >> >> requirements enforcement via checks rather than explicit use of a >> >> restricted mirror. There are some things to sort out like platform >> >> dependent wheels (I am not sure that any OpenStack project directly >> >> consumes these but I have found them to be quite handy) and the >> >> potential need for more enforcement to keep this working, but I think >> >> this is a possibility. >> > >> > This would be neat. >> > >> > -Angus >> > >> > >> >> Clark >> > >> >> [2] >> >>> >> >>> https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/slave_scripts/select-mirror.sh#n54 >> > >> >> _______________________________________________ >> >> OpenStack-dev mailing list >> >> OpenStack-dev@lists.openstack.org >> >>> <mailto:OpenStack-dev@lists.openstack.org> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> > >> >> > >> >> > _______________________________________________ >> >> > OpenStack-dev mailing list >> >> > OpenStack-dev@lists.openstack.org >> >> <mailto:OpenStack-dev@lists.openstack.org> >> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/ >> >> >> >> _______________________________________________ >> >> OpenStack-dev mailing list >> >> OpenStack-dev@lists.openstack.org >> >> <mailto:OpenStack-dev@lists.openstack.org> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> > >> > >> > _______________________________________________ >> > OpenStack-dev mailing list >> > OpenStack-dev@lists.openstack.org >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >> >> -- >> Sean Dague >> http://dague.net >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
