On Sun 2017-09-17 16:26:25 -0700, Russ Allbery wrote:
> I personally lean towards 2, which is consistent with what's in Policy
> right now, but I can see definite merits in 3. I believe the reproducible
> builds project is currently sort of doing 1, but I have a hard time seeing
> how to make that viable on the testing side.
Thanks for raising this question, Russ!
I'm not sure that we should let lack of exhaustive testing push us away
from (1). (1) is in principle the right thing -- it's easy to make a
build reproducible if we tell people that they have to do exactly one
specific thing. But we generally want people to be able to run
heterogenous systems, and not to force them into one particular
environment.
Consider someone who wants to see more logging from a build, for
example. There could be an environment variable that encourages the
toolchain to log more, but doesn't affect the binary objects created by
the build. By going with choices (2) or (3) we effectively dismiss even
considering the reproducibility of those builds, which seems like a
shame.
Does everything in policy need to be rigorously testable? or is it ok
to have Policy state the desired outcome even if we don't know how (or
don't have the resources) to test it fully today.
I'd prefer for policy to be able to make strong advisory statements even
without us being able to test them mechanically. This is already the
case for (for example) "preferred form of modification" -- it's partly
testable, but will never be 100% testable, and will always require
research and discussion and thinking for the corner cases. Yet we
continue to aim for it.
Policy should be aiming high, not lowering the bar to meet what's
concretely testable.
--dkg
