Hi, snapshot.debian.org is an awesome service for the wider free software community and especially for those working on reproducible builds. Sadly accessing *many* packages from it is limited and troublesome (see below for bug numbers), thus we (mostly Frédéric Pierret and myself) came up with the idea of setting up a partial mirror, covering only the years 2017 until now and arch:amd64 and arch:all only as well. (for a start, maybe we need 2015+2016 too and maybe we can afford to also host arm64 or some other architecture...)
Background: several projects (at NYU, from Qubes, from Debian, some independent
researchers) want to setup rebuilders of Debian (bullseye) rebuilding all 30000
source packages which need thousands of different snapshots and thus are
regularily
hit/hurt by these bugs:
#977653 Please document rate limits on snapshots.debian.org
#960304 snapshot.debian.org: Snapshot repo repeatedly cutting off connection,
returning partial content
#969906 snapshot.debian.org: error 500 internal server error after some
requests via Python
It's possible to work around them partly/sometimes, but it seems clear by
now that using snapshot.d.o *as it is* doesn't scale for our use cases. Hence
the idea of a partial snapshot mirror which is only suitable to rebuild bullseye
am64 (but not previous release or other archs.) - and which can also be mirrored
more easily than the whole of snapshot.d.o if the need arises.
Does that sound feasable/sensible from the snapshot.d.o team perspective?
The idea would be to ask Debian for money to buy four 16tb discs (one currently
costs around 300€) and then have two shipped to Frédéric (for local development)
and two shipped to OSUOSL.org, which probably could host them for us. (We've got
an informal offer we need to formalize once/if this idea is deemed good.)
misc notes:
this is ment as prototype / temporary solution for the next 2-3 years to that
we can
continue to develop (several) tools for the verification of Reproducible Builds
of
amd64/bullseye and probably amd64/bookworm, but it's clear that in the long
term we want solutions for other supported archs and releases after bookworm.
I'd very much hope that snapshot.d.o can be scaled, just this will take time,
probably
a lot, and so I think it makes sense to spend <1.5k€ to have an interim
solution *now*.
In theory the machine offered to us informally at osuosl can take 6 sata
drives...
feedback very much welcome!
--
cheers,
Holger
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ PGP fingerprint: B8BF 5413 7B09 D35C F026 FE9D 091A B856 069A AA1C
⠈⠳⣄
I'm looking forward to Corona being a beer again and Donald a duck.
signature.asc
Description: PGP signature
_______________________________________________ Reproducible-builds mailing list Reproducible-builds@alioth-lists.debian.net https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/reproducible-builds
