w00t! thanks infra team On Wed, Feb 10, 2016 at 7:45 PM, Monty Taylor <mord...@inaugust.com> wrote: > Hey everybody, > > tl;dr - We have new AFS-based consistent per-region mirrors of PyPI and APT > repos with additional wheel repos containing pre-built wheels for all the > modules in global-requirements > > We've just rolled out a new change that you should mostly never notice - > except that jobs should be a bit faster and more reliable. > > The underpinning of the new mirrors is AFS, which is a global distributed > filesystem developed by Carnegie Mellon back in the 1980's. In a lovely fit > of old-is-new-again, the challenges that software had to deal with in the > 80s (flaky networks, frequent computer failures) mirror life in the cloud > pretty nicely, and the engineering work to solve them winds up being quite > relevant. > > One of the nice things we get from AFS is the ability to do atomic > consistent releases of new filesystem snapshots to read-only replicas. That > means we can build a new version of our mirror content, check it for > consistency, and then release it for consumption to all of the consumers at > the same time. That's important for the gate, because our "package not > found" errors are usually about the mirror state shifting during a test job > run. > > We've had per-region PyPI mirrors for quite some time (and indeed the gate > would largely be dead in the water without them). The improvement from this > work for them is that they're now AFS based, so we should never have a > visible mirror state that's wonky or inconsistent between regions, and we > can more easily expand into new cloud regions. > > We've added per-region apt mirrors (with yum to come soon) to the mix based > on the same concept - we build the new mirror state then release it. There > is one additional way that apt can fail even with consistent mirror states, > which is that apt repos purge old versions of packages that are no longer > referenced. If a new mirror state rolls out between the time devstack runs > apt-get update and the time it tries to do apt-get install of something, you > can get a situation where apt is trying to install a version of a package > that is no longer present in the archive. To mitigate this, we're purging > our mirror on a delay ... in our mirror runs every 2 hours we add new > packages and update the index, and then in the next mirror run we'll delete > the packages the previous run made unreferenced. This should make apt errors > about package not found go away. > > Last but certainly not least, there are now also wheel repositories of > wheels built for all of our python packages from global-requirements. This > is a speed increase and shaves 1.8 tens of minutes off of a normal devstack > run. > > With these changes, it means we're writing not only pip.conf but now > sources.list files into the test nodes. If you happen to be doing extra > special things with either of those in your jobs, you'll want to make sure > you consume the config files we're laying down > > Finally, although all Infra projects are a team effort - a big shout out to > Michael Krotschek and Jim Blair for diving in and getting this finished over > the past couple of weeks. > > Monty > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-- Davanum Srinivas :: https://twitter.com/dims __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
