Hi Ludovic,
l...@gnu.org (Ludovic Courtès) writes:
> Mark H Weaver <m...@netris.org> skribis:
>
>> If I'm not mistaken, I believe I have confirmed with the test below that
>> a substitute for binutils from early commencement on aarch64 is not
>> available on berlin.
>
> [...]
>
>> It occurs to me that on Hydra, I have implemented a system to ensure
>> that *all* derivations from a certain set of _evaluations_ (the most
>> recent release and the last two weeks of 'master' evaluations) are
>> permanently kept as GC roots, regardless of how long ago they were
>> built. Among other things, this ensures that even the early
>> commencement derivations are kept even if they were built a long time
>> ago.
>>
>> Berlin.guixsd.org may not have any similar mechanism in place. It may
>> still be using the old policy, where old GC roots are deleted based
>> solely on their timestamps in the filesystem, which indicate when the GC
>> root symlinks were first installed, long ago during the last core
>> updates build-out. This could result in the early commencement
>> derivations and corresponding outputs being deleted prematurely.
>
> Correct. Berlin uses ‘guix publish’ with a TTL of 45 days: if a nar is
> not requested within 45 days, and if its corresponding store item was
> GC’d in the meantime, it disappears.
The 'guix publish' TTL is a secondary issue, because, as you say, the
NARs are only deleted if the corresponding store item has been GC'd.
The more important question is: what is the policy for deleting GC roots
on Berlin?
Regardless, I think we should seriously consider moving the Aarch64
build slave(s) to Hydra for now, until Cuirass is more mature.
That Aarch64 is only supported on Berlin causes several problems:
* We're unable to effectively monitor when packages become broken on
Aarch64, due to the lack of a mature web interface. I'm glad to see
that progress is being made there, but it's still very far from
sufficient for our purposes.
* When builds fail, they cannot be restarted on Berlin. It is quite
common for faulty test suites or other problems to cause spurious
failures, sometimes leading to a many important dependency failures.
This leads to missing substitutes.
* Our small number of Aarch64 build slaves makes it crucial to
prioritize the most important builds. On Hydra, I regularly inspect
the build queue and prioritize builds that I deem to be important. I
also cancel builds when appropriate, e.g. for wip branches such as
staging or core-updates, when a new evaluation is made, I cancel the
outdated builds. These interventions are quite important in practice,
especially for the slower architectures, or architectures with
insufficient build capacity, because we often create new builds faster
than our build farm is able to sustain.
* As we've just discovered, substitutes for early commencement packages,
and possibly other important packages, are missing on Aarch64 due to
an inadequate GC policy on Berlin, which makes Aarch64 effectively
unusable.
What do you think?
Mark
