On Wed, 13 Apr 2016, Bjoern Nyjorden wrote:
> Given that this is not the first occurrence,
I think it is, actually. As often is the case in the swiss-cheese
model, here all the holes lined up and the update of this security
mirror was delayed for about two days.
We can identify at least four causal factors. Probably more, if we
look a bit further.
(1) The scripts Debian uses to mirror repositories treat the mirroring
hierarchy as a tree. The failure of any node or link will cause
the subtrey(s) under the failed component to not receive updates.
(2) There is an ongoing network outage between where the australian
mirror is and its upstream mirror in the US.
(3) The scripts that automatically update the security rotation only
check if a server is online and responds to http requests - it
does not check if a mirror is current.
(4) The nagios warning was missed in all the noise, and the relevant
teams are overworked and busy.
> what options does the Debian
> community have available to prevent this problem from arising again in the
> future?
Fixing (1) would be really nice. It requires somebody to sit down and
design and implement something better.
Re (2), we are temporarily syncing the .au mirror from a different
machine while the network outage is being traced down by the relevant
NOCs.
Expanding mini-nag[1] to do something about (3) would be nice too.
Cheers,
[1]
https://anonscm.debian.org/cgit/mirror/dsa-mini-nag.git/tree/
also see
https://anonscm.debian.org/cgit/mirror/dsa-auto-dns.git/tree/
--
| .''`. ** Debian **
Peter Palfrader | : :' : The universal
https://www.palfrader.org/ | `. `' Operating System
| `- https://www.debian.org/
