On Sun, Jul 13, 2014, at 02:36 PM, James Polley wrote:
We're also thinking about how we continue to offer the pre-built wheels for each of our build platforms. For infra, what I'm thinking is: On each mirror slave (We have one for each OS combo we use), do something similar to: pip wheel -r global-requirements.txt rsync $wheelhouse [1]pypi.openstack.org/$(lsb_release) This may require keeping pypi-mirror and using an option to only do wheel building so that we can get the directory publication split. Ok. I got bored and wrote that: [2]https://review.openstack.org/106638 So if we land that, you can do; pip wheel -r global-requirements.txt run-mirror --wheels-only --wheelhouse=wheelhouse --wheeldest=mirror rsync -avz mirror pypi.openstack.org:/srv/mirror If we went the devpi route, we could do; pip wheel -r global-requirements.txt for pkg in $wheelhouse; do devpi upload $pkg done And put that into a cron. Obviously "keeping pypi-mirror" would require the least amount of change to how we suggest developers set up their systems. I think the devpi option seems fairly reasonable too. It looks like it's easier (and faster, and less bandwidth-consuming) than setting up bandersnatch or apt-mirror, which we currently suggest people consider. It doesn't look any more heavyweight than having a squid proxy for caching, which we currently suggest as a bare minimum. For an individual dev testing their own setup, I think we need a slightly different approach from the infra approach listed above though. I'm assuming that it's possible to probe the package index to determine if a wheel is available for a particular version of a package yet. If that's the case, we should be able to tweak tools like os-svc-install to notice when no wheel is available, and build and upload the wheel. I think this should give us a good balance between making sure that each build (except the first) uses wheels to save time, still gets the latest packages (since the last time the system was online at least), and the user doesn't need to remember to manually update the wheels when they're online. This gave me an idea: There was talk about pip being able to use a wheel cache (wheelhouse). Can we bind-mount an arch-specific wheelhouse from the hypervisor into our chroots as we build? This would let people get most of the wheel speedup while doing almost no specifal configuration. -Greg References 1. http://pypi.openstack.org/$(lsb_release) 2. https://review.openstack.org/106638
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
