Antero Mejr <m...@antr.me> writes: > I am looking for feedback/sponsors for a new GCD proposal to set up one > or more international (outside EU) binary substitute servers in order to > improve download speeds. > > The PR is not ready yet because the proposal needs to be a more concrete > plan. I would like to collect people's feedback before updating it - if > you all think this issue is important, whether colocation vs. cloud > would be preferable, where the servers should be located, or any other > comments/concerns. > > The working repository is here: > https://codeberg.org/antero/guix-consensus-documents/src/branch/wip-substitutes > > Here is the initial markdown text: > > title: International substitute servers > id: 6 > status: draft > discussion: <draft|submitted|accepted|withdrawn|deprecated> > authors: Antero Mejr > sponsors: <Sponsor Name> > date: <date when the discussion period starts> > draft-date: 2025-08-07 > discussion-date: <date when the discussion period starts> > deliberation-date: <date when the deliberation starts> > SPDX-License-Identifier: CC-BY-SA-4.0 OR GFDL-1.3-no-invariants-or-later > --- > > # Summary > > The official binary substitute servers for Guix, `ci.guix.gnu.org` and > `bordeaux`, are both located in Western Europe. Users outside Western Europe > may > experience slower speeds when downloading substitutes: 50-200 kbps is not > uncommon. Since substitutes can be quite large, this causes significant slow > down when performing package operations. This RFC proposes the establishment > of > additional substitute servers in other regions. > > # Motivation > > In the present state, users outside Europe experience substitute download > speeds > on the order of kilobits per second. This comprises some of the perceived > "slowness" of Guix, as mentioned in all three of the [Guix > surveys](https://guix.gnu.org/en/blog/2025/guix-user-and-contributor-survey-2024-the-results-part-1/). > Other package managers with distributed substitute servers achieve download > speeds of tens or even hundreds of megabytes per second, many times faster > than > the single Guix substitute server can provide. > > Common dependency packages such as LLVM can be large, around 50-200 MB. > Downloading substitutes for them on such a slow connection will take at least > 5 > minutes, and that is only for a single dependency. This is a significant > impediment to package management operations and workflows. > > Distributing substitutes can also increase reliability. As it stands, if > `ci.guix.gnu.org` is not available, the user will not be able to download > substitutes at all. It also assists with reproducibility: more identical > substitute builds means a higher degree of assurance of deterministic, > reproducible packages.
https://qa.guix.gnu.org/reproducible-builds Above link is reproduciblity statistics for the two existing build farms: around 1k packages are known unreproducible, while over 20k packages remain unknown (build not finished on both build farms). This means adding another independent build farm will offer limited help regarding reproduciblity, instead putting efforts into existing issues and build farms would be a better approach, which is also mentioned in the manual: Contributing to Guix’s Infrastructure https://guix.gnu.org/manual/devel/en/html_node/Contributing-to-Guix_0027s-Infrastructure.html I think you were referring to "substitute servers" as both "build farms" and "mirrors" so I'll be more explicit here. I have expressed my opinion on "build farms" above, and I think there's more need for a direction on "mirrors". Regarding mirrors, I think the best option is to add guix into existing public mirrors, since they have more resource and experience in operating mirrors. Popular public mirrors may also have downstreams, improving the coverage further. Below are existing attempts I can find: Can TUNA mirror GuixSD to mainland China? https://github.com/tuna/issues/issues/199 Can USTC mirror GuixSD to mainland China? https://github.com/ustclug/mirrorrequest/issues/96 Mirror request: Guix https://github.com/sjtug/mirror-requests/issues/131 A mirror for Guix? https://github.com/PhirePhly/micromirrors/issues/1 Only one of them are accepted, with a special setup that can't be implemented by most. I learned more about this topic when I became the one to send the request: Substitutes mirror available in Japan! (and questions regarding mirroring) https://lists.gnu.org/archive/html/guix-devel/2025-05/msg00043.html This mirror is currently down, btw: Slow substitute narinfo querying for japanese mirror https://lists.gnu.org/archive/html/help-guix/2025-08/msg00081.html nar-herder can address the performance issue for reverse proxy, but most public mirrors may just reject the special setup. Unless we're going to host all the mirrors by ourselves, we need to have a goal on supporting rsync mirroring without extra setup. I'm going to patch guix publish specially for the build farm I run, btw, as a temporary solution. I'll announce it on the list when things are settled. Also there can be another question, since we have two independent build farms, I don't see a reason to mirror both if not using our infrastructure, then which one should be mirrored? > > # Detailed Design > > Many package managers make available dozens of substitute servers (often > called > "mirrors") from which to download compiled packages. Usually package managers > have some sort of configuration option to select the mirror with the fastest > connection, typically the mirror that is geographically closest, and possibly > some sort of auto-detection of connection speeds. Guix does not currently have > such a configuration option, and defaults to `ci.guix.gnu.org`. The addition > of > a mirror selection configuration option is part of this proposal. Such an > option > could be a `guix-configuration` option of `guix-service-type`. > > The Guix installer, for both the ISO and foreign-distro forms, would have an > additional prompt to select a substitute server from a list. > > Since many Guix users and developers are based in the United States, a > US-based > server would be a sensible first step beyond the European region. From there > community feedback can be solicited to determine locations for future servers, > so long as sufficient funds are available. > > ## Cost of Reverting > > Cost scale: 0 - No incompatibility > > Users of Guix System would experience no change, as substitute server > authentication is handled automatically. Users of Guix on foreign distros who > are installing for the first time would experience no change, so long as they > choose to authorize binary substitutes at install time. > > For users of Guix that is already installed on a foreign distro, they may need > to follow [these > steps](https://guix.gnu.org/manual/en/html_node/Substitute-Server-Authorization.html) > to authorize the new substitute servers if they wish to use them. Otherwise > they > would not see the benefits. > > No deprecation of existing substitute servers would be necessary. > > Depending on whether colocation is used, maintenance of the substitute server > may require a Guix developer in the geographic area of the server, with proper > vetting/access clearance to perform administrative tasks directly on the > hardware. Remote maintenance of the server(s) would be possible for most > operations. > > Should a new substitute server need to be decommissioned (for whatever > reason), > there would need to be a process established for doing so without breaking > compatibility. This would involve having the decommissioned substitute server > URL redirect to a different active and authorized substitute server. > > # Drawbacks and Open Issues > > The main drawback of maintaining additional substitute servers is cost, in > both > the initial investment and recurring costs. For example, Nix serves their > infrastructure through [Amazon > AWS](https://discourse.nixos.org/t/s3-sponsorship-extension-more-resources-to-build-a-more-sustainable-nix/50936) > and their recurring costs are $15,000 per month. > > However, Guix could potentially mitigate some of the costs. Making use of > donated hardware, and using colocation instead of the cloud could reduce the > monthly spend. Also the FSF may have resources that can provide servers or > colocation services at a reduced price. Guix also has an FSF [Working Together > Fund](https://my.fsf.org/civicrm/contribute/transact?reset=1&id=50) that could > potentially be used to fund substitute servers. > > The [Guix Foundation](https://foundation.guix.info/) may also be interested in > providing financial support for international substitute servers.
