I'm in!
Ludovic Courtès <l...@gnu.org> writes:
> Hello Guix!
>
> This week, the node behind {ci,issues,disarchive}.guix.gnu.org and
> guix.gnu.org was down twice for a few hours—nothing terrible in the end,
> but it reminded us that, even though Guix doesn’t rely on any particular
> machine, we can definitely feel the inconvenience when it’s down.
>
> We were unlucky enough that it happened days after the other build farm,
> bordeaux.guix.gnu.org, ran out of disk space and had its CI stopped,
> right before the big merge—so it doesn’t have substitutes for current
> master.
>
> While discussing this on IRC the other day, we thought that perhaps it
> was time to have an infrastructure hackathon. How about Tuesday,
> Dec. 21st? (Probably with a followup in January.)
>
> Here are tasks that were brought up:
>
> • Set up a backup server for berlin.guix.gnu.org, the head node of the
> ci.guix.gnu.org, possibly moving some services such as the web site
> there.
>
> • Add DNS redundancy for guix.gnu.org so it can point to one of two
> hosts (need to figure out certbot challenges so both machines can
> update their certificates).
>
> • Set up status.guix.gnu.org with sysadmin status updates (possibly
> using Prometheus?).
>
> • Come up with a plan to add disks to the RAID array on bayfront, the
> head node of bordeaux.guix.gnu.org.
>
> • Work on a plan to back up the Disarchive database currently on
> berlin.guix.
>
> • Work on a plan to mirror nars from ci.guix and bordeaux.guix, using
> plain rsync or <https://git.cbaines.net/guix/nar-herder/about/>.
>
> • Have a documented procedure to set up substitute mirrors, such as
> the one in .cn (I can’t find the URL), ideally with plain rsync
> access.
>
> Am I forgetting something?
>
> Some of these tasks require root or physical access for the final steps,
> but most of them are about (1) coming up with a plan, and (2) adjusting
> the system configuration at
> <https://git.savannah.gnu.org/cgit/guix/maintenance.git/tree/hydra/>.
> Anyone with sysadmin experience to share and Guix System knowledge can
> join! We’ll communicate over #guix on irc.libera.chat.
>
> Who’s in? :-)
>
> Cheers,
> Ludo’.
