Hey everybody,
tl;dr - We have new AFS-based consistent per-region mirrors of PyPI and
APT repos with additional wheel repos containing pre-built wheels for
all the modules in global-requirements
We've just rolled out a new change that you should mostly never notice -
except that jobs should be a bit faster and more reliable.
The underpinning of the new mirrors is AFS, which is a global
distributed filesystem developed by Carnegie Mellon back in the 1980's.
In a lovely fit of old-is-new-again, the challenges that software had to
deal with in the 80s (flaky networks, frequent computer failures) mirror
life in the cloud pretty nicely, and the engineering work to solve them
winds up being quite relevant.
One of the nice things we get from AFS is the ability to do atomic
consistent releases of new filesystem snapshots to read-only replicas.
That means we can build a new version of our mirror content, check it
for consistency, and then release it for consumption to all of the
consumers at the same time. That's important for the gate, because our
"package not found" errors are usually about the mirror state shifting
during a test job run.
We've had per-region PyPI mirrors for quite some time (and indeed the
gate would largely be dead in the water without them). The improvement
from this work for them is that they're now AFS based, so we should
never have a visible mirror state that's wonky or inconsistent between
regions, and we can more easily expand into new cloud regions.
We've added per-region apt mirrors (with yum to come soon) to the mix
based on the same concept - we build the new mirror state then release
it. There is one additional way that apt can fail even with consistent
mirror states, which is that apt repos purge old versions of packages
that are no longer referenced. If a new mirror state rolls out between
the time devstack runs apt-get update and the time it tries to do
apt-get install of something, you can get a situation where apt is
trying to install a version of a package that is no longer present in
the archive. To mitigate this, we're purging our mirror on a delay ...
in our mirror runs every 2 hours we add new packages and update the
index, and then in the next mirror run we'll delete the packages the
previous run made unreferenced. This should make apt errors about
package not found go away.
Last but certainly not least, there are now also wheel repositories of
wheels built for all of our python packages from global-requirements.
This is a speed increase and shaves 1.8 tens of minutes off of a normal
devstack run.
With these changes, it means we're writing not only pip.conf but now
sources.list files into the test nodes. If you happen to be doing extra
special things with either of those in your jobs, you'll want to make
sure you consume the config files we're laying down
Finally, although all Infra projects are a team effort - a big shout out
to Michael Krotschek and Jim Blair for diving in and getting this
finished over the past couple of weeks.
Monty
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
