On Tue, Jun 24, 2014 at 9:54 PM, Adrian Otto <adrian.o...@rackspace.com> wrote: > Hello, > > Solum has run into a constraint with the current scheme for requirements > management within the OpenStack CI system. We have a proposal for dealing > with this constraint that involves making a contribution to openstack-infra. > This message explains the constraint, and our proposal for addressing it. > > == Background == > > OpenStack uses a list of global requirements in the requirements repo[1], and > each project has it’s own requirements.txt and test-requirements.txt files. > The requirements are satisfied by gate jobs using pip configured to use the > pypi.openstack.org mirror, which is periodically updated with new content > from pypi.python.org. One motivation for doing this is that pypi.python.org > may not be as fast or as reliable as a local mirror. The gate/check jobs for > the projects use the OpenStack internal pypi mirror to ensure stability. > > The OpenStack CI system will sync up the requirements across all the official > projects and will create reviews in the participating projects for any > mis-matches. Solum is one of these projects, and enjoys this feature. > > Another motivation is so that users of OpenStack will have one single set of > python package requirements/dependencies to install and run the individual > OpenStack components. > > == Problem == > > Stackforge projects listed in openstack/requirements/projects.txt that decide > to depend on each other (for example, Solum wanting to list mistralclient as > a requirement) are unable to, because they are not yet integrated, and are > not listed in openstack/requirements/global-requirements.txt yet. This means > that in order to depend on each other, a project must withdraw from > projects.txt and begin using pip with pypi.poython.org to satisfy all of > their requirements.I strongly dislike this option. > > Mistral is still evolving rapidly, and we don’t think it makes sense for them > to pursue integration wight now. The upstream distributions who include > packages to support OpenStack will also prefer not to deal with a requirement > that will be cutting a new version every week or two in order to satisfy > evolving needs as Solum and other consumers of Mistral help refine how it > works. > > == Proposal == > > We want the best of both worlds. We want the freedom to innovate and use new > software for a limited selection of stackforge projects, and still use the > OpenStack pypi server to satisfy my regular requirements. We want the speed > and reliability of using our local mirror, and users of Solum to use a > matching set of requirements for all the things that we use, and integrated > projects use. We want to continue getting the reviews that bring us up to > date with new requirements versions. > > We propose that we submit an enhancement to the gate/check job setup that > will: > > 1) Begin (as it does today) by satisfying global-requirements.txt and my > local project’s requirements.txt and test-requirements.txt using the local > OpenStack pypi mirror. > 2) After all requirements are satisfied, check the name of my project. If it > begins with ‘stackforge/‘ then look for a stackforge-requirements.txt file. > If one exists, reconfigure pip to switch to use pypi.python.org, and satisfy > the requirements listed in the file. We will list mistralclient there, and > get the latest tagged/released version of that. > I am reasonably sure that if you remove yourself from the openstack/requirements project list this is basically how it will work. Pip is configured to use the OpenStack mirror and fall back on pypi.python.org for packages not available on the OpenStack mirror [2]. So I don't think there is any work to do here with additional requirements files. It should just work. Adding a new requirements file will just make things more confusing for packagers and consumers of your software. > > == Call To Action == > > What do you think of this approach to satisfy a balance of interests? > Everything remains the same for OpenStack projects, and Stackforge projects > get a new feature that allows them to require software that has not yet been > integrated. Are there even better options that we should consider? > > Thanks, > > Adrian Otto > > > References: > [1] https://review.openstack.org/openstack/requirements
For what it is worth the Infra team has also been looking at potentially using something like bandersnatch to mirror all of pypi which is now a possibility because OpenStack doesn't depend on packages that are hosted external to pypi. We would then do requirements enforcement via checks rather than explicit use of a restricted mirror. There are some things to sort out like platform dependent wheels (I am not sure that any OpenStack project directly consumes these but I have found them to be quite handy) and the potential need for more enforcement to keep this working, but I think this is a possibility. Clark [2] https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/slave_scripts/select-mirror.sh#n54 _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
